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Executive  Summary 

This  report  summarizes  the  research  activities  undertaken  as  part  of  the  "Effects  of  Cognitive 
Load  on  Trust"  project  in  conjunction  with  the  US  AFRL  and  Sunway  University.  NICTA's 
role  comprised  the  measurement  and  assessment  of  cognitive  load  through  speech  and  other 
interaction  modalities.  The  project  initially  involved  examining  whether  the  relationship 
between  cognitive  load  and  trust  judgments  had  been  examined  in  the  literature.  A  synthesis 
of  this  effort  is  detailed  in  the  first  section  of  this  report. 

The  second  major  milestone  in  this  project  was  the  completion  of  the  planning  and 
design  of  the  user  study  for  all  three  sites  (US,  Malaysia  and  Australia),  including 
coordinating  multi-national  IRB  approval  for  the  experiments.  The  third  milestone  involved 
the  implementation  of  the  software  for  data  collection,  and  the  production  of  supplementary 
materials  to  be  used  in  the  study.  The  outcomes  for  this  part  of  the  work  are  described  in  the 
second  section  of  the  report.  The  execution  of  the  study  was  divided  into  three  data  collection 
phases,  one  for  each  site  (Australia,  US  and  Malaysia).  The  Australian  user  study  has  been 
completed,  with  approximate  100  university  students  sampled  and  analysis  is  underway.  A 
summary  of  the  data  collection  outcomes  so  far  is  also  included  here. 
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1.  Introduction 

Trust  is  found  to  be  a  critical  factor  driving  human  behavior  in  both  interpersonal  and 
computer-based  interactions.  Previous  research  by  Mayer  et  al.  [1]  has  found  three 
trustworthiness  elements  that  influence  the  development  of  trust  in  interpersonal  situations: 
ability,  benevolence,  and  integrity.  Thus  far,  only  a  few  studies  have  looked  at  how  different 
situational  factors  influence  trust  development  as  reflected  in  the  relative  salience  of  the  three 
trustworthiness  indicators.  One  dominant  situational  factor  that  may  shape  trust  perceptions 
of  an  information  source  is  culture.  Similarly,  little  is  known  how  cognitive  load  may  affect 
the  different  trustworthiness  factors  during  trust  development  and  acquisition. 

The  3-year  research  project  proposed  serves  as  part  of  a  larger  international  research  effort  in 
collaboration  with  Dr.  Lyons  and  Dr.  Stokes  (AFRL),  and  Dr.  Yeo  (University  of  Malaysia 
Sarawak),  with  separate  proposals  to  be  submitted  through  the  AFOSR/AOARD  programs. 
A  three-part  user  experiment  was  designed  -  one  in  the  US,  one  in  Australia,  and  one  in 
Malaysia,  to  investigate  the  cross-cultural  influences  on  trust.  The  Australian  part  of  the  data 
collection  has  been  completed  thus  far,  with  the  US  and  Malaysian  counterparts  currently  in 
progress. 

2.  Project  Plan  Updates 

The  following  project  plan  was  agreed  to  as  part  of  the  grant  approval  for  the  first  year  of  this 
project.  No  significant  amendments  have  been  made  to  this  initial  plan.  At  the  end  of  the  first 
year,  milestones  Ml-  M4  are  complete,  while  M5  is  in  progress  and  the  project  is  running  on 
time.  This  document  details  our  progress. 


ID 

Milestone 

Deliverable/Outcome 

Due  Date 

Ml 

Complete  Pre-Pilots  (Materials) 

•  Pilot  test  the  neutrality  of  the 
stimulus  data  to  be  used  in  the 
experiment 

•  Stimulus  material  in  target 
demographic  (Australian) 

•  Make  changes  to  the  stimulus 
material  as  appropriate  to  ensure 
neutrality 

Jul  30,  2011 

M2 

Experiment  Tool  Design 

•  Development  of  the  experimental 
application  to  be  used 

•  Implement  factor  manipulations, 
including  cognitive  load 

•  Implement  data  collection 
functionality  as  part  of  the  design 

Aug  31, 

2011 

M3 

Complete  Pre-Pilots  (Study 

Design) 

•  Conduct  pilots  on  target 
demographic  (6  participants) 

•  Evaluate  study  design, 
procedure,  physical  set-up 

Sep  30, 

2011 
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•  Assess  changes  needed  at  each 
site 

M4 

Complete  Experimental  Study 

•  Source  participants 

•  Run  the  study 

•  Debrief  participants 

Nov  30, 

2011 

M5 

Signal  Analysis  of  Speech  Data 

•  Collect  speech  data  from  other 
sites 

•  Segment,  annotate  and  label 
speech  data 

•  Build  speech  models  to  represent 
cognitive  load  levels. 

•  Report  results  to  rest  of  the  team 

Jan  31, 

2012 

M6 

Linguistic  Analysis  of  Speech 

Data 

•  Prepare  speech  annotations 

•  Run  linguistic  analyses  on  text 
data  derived  from  speech 

Mar  31, 

2012 

M  7 

Consolidate  Speech/Linguistic 
Findings 

•  Ground  truth  analysis  (subjective 
ratings,  performance) 

•  Contextualise  the  findings  with 
those  from  Trust  based 
manipulations,  looking  for 
interaction  effects 

Jun  30, 

2012 

M8 

Project  Management 

•  Weekly  meetings 

•  Team  workshops,  including 
conference  calls  with  co¬ 
investigators 

•  Year-end  final  report  circulated  to 
AOARD  office  and  all  other 
investigators 

Jun  30, 

2012 

3.  Literature  Review 

The  proposed  research  will  evaluate  the  relative  strength  of  Mayer's  trustworthiness 
indicators  (ability,  benevolence,  and  integrity)  in  both  a  collectivistic  and  individualistic 
culture.  To  date,  there  have  been  no  such  studies  that  have  empirically  tested  this  model  in 
different  cultures.  Additionally,  the  research  will  examine  the  relationship  between  cognitive 
load  (CL)  and  trust  characterized  by  the  trustworthiness  indicators,  and  whether  this  CL-trust 
relationship  is  affected  by  cultural  factor. 

Cognitive  load  and  trust:  Cognitive  load  is  a  key  component  of  the  four-stage  model  of  human 
information  processing  [8],  and  it  is  clear  that,  like  trust,  it  plays  an  important  role  in 
mediating  human  behavior  during  collaboration  with  other  humans  or  automated  systems. 
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Despite  this,  little  is  known  about  the  relationship  between  the  two  in  such  contexts. 
Intuitively  we  might  guess  that  as  cognitive  load  increases,  a  person  may  choose  to  rely  more 
heavily  on  colleagues  or  an  automatic  system  and  in  the  process  display  more  indicators  of  an 
implicit  trust  of  the  system.  Alternatively,  under  higher  load  the  person  may  be  reticent 
increase  their  trust  and  instead  adapt  their  strategy  to  manage  the  increased  task  complexity 
to  avoid  increased  dependence  on  others  or  on  an  automatic  system.  Although  the  authors 
note  that  the  empirical  evidence  is  limited,  Parasuraman  and  Riley  [9]  argue  that  increased 
workload  is  often  cited  as  one  of  the  most  important  factors  in  choice  to  use  automation. 
However,  in  a  comparison  of  trust  in  various  levels  of  automation,  including  manual  control, 
Ruff  and  colleagues  [10]  found  that  as  workload  increased,  subjective  reports  of  trust 
decreased  for  automation,  but  increased  for  manual  control.  In  an  explanation  for  the 
equivocal  evidence  for  trust  and  automation  use,  Parasuraman  and  Riley  [9]  suggest  that 
complex  task  domains  may  prompt  different  task  strategies,  such  as  use  of  automation 
during  high  cognitive  load  even  if  trust  is  low.  Thus,  under  high  cognitive  load,  use  of 
automation  and  trust  (subjectively  measured)  may  not  be  aligned,  as  is  often  assumed.  The 
proposed  study  seeks  to  clarify  the  equivocal  results  of  previous  research  through  the 
examination  of  the  fundamental  trust  process  (i.e.,  not  simply  use  of  automation)  under 
varying  levels  of  cognitive  load. 

Previous  investigation  of  the  effect  of  cognitive  load  in  user  interfaces  suggests  the 
entrenchment  of  established  behaviors  with  increasing  load  [11].  Other  research  also 
suggests  that  with  increased  cognitive  load  users  revert  to  older,  over  learned  and  simpler 
types  of  responses  [12].  When  a  user  has  a  pre-existing  trust  of  an  automated  system,  the 
implication  is  that  they  will  tend  to  over-trust  the  system  during  higher  cognitive  load  [13], 
and  a  similar  result  might  be  expected  of  another  human  agent  rather  than  a  system.  The 
Affect  Infusion  Model  [14]  suggests  that  during  faster  processing,  individuals  use  their 
affective  states  as  a  short-cut  to  infer  their  evaluative  reactions  to  a  target,  such  as  in 
judgments  of  trust.  Thus,  it  is  likely  that  similar  reliance  on  established  cultural  values  and 
attitudes  may  rise  to  the  surface  during  higher  cognitive  loads.  With  respect  to  the  elements 
of  trustworthiness,  it  has  been  found  that  action-oriented  and  performance-oriented  cultures 
put  more  value  on  a  party's  ability,  while  collaborative  and  relationship-oriented  cultures 
emphasise  more  on  the  party's  benevolence. 

Based  on  the  theories  proposed  by  Lee  and  See  [15],  and  Mayer  et  al.  [1],  an  initial  conceptual 
model  of  trust  developed  by  Dr.  Stokes  (AFRL)  for  our  further  research  is  shown  in  Figure  1. 
In  this  model,  trust  is  an  attitude  that  drives  behavior  through  intention.  While  trust  is 
affected  by  the  trustworthiness  indicators  perceived  under  the  influence  of  various  cognitive 
load  levels,  at  the  same  time,  cognitive  load  is  affecting  behavior  as  a  main  cognitive 
constraint  using  competing  resource  in  the  mental  space.  Due  to  the  error  feedback  loop  of 
past  experience,  trust  can  also  be  affected  dynamically  by  the  fluctuation  in  cognitive  load. 
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Inputs  Beliefs  &  Perceptions  Attitudes  Intention  Behavior 


Figure  1:  Proposed  conceptual  model  of  trust 

Based  on  the  literature  review  conducted  by  the  team,  to  the  best  of  our  knowledge,  there  are 
no  prior  studies  that  attempted  to  investigate  any  relationship  between  trust  and  the 
construct  of  cognitive  load  in  detail.  A  few  studies  have  attempted  to  isolate  the  effects  of 
high  workload  and  stress  on  the  level  of  people's  trust  judgements  but  their  focus  was  more 
on  the  people's  trust  perception  of  organisations  and  of  automation  systems,  e.g.  Mayer  and 
Davis  (1999)  [2],  and  Biros,  Daly,  and  Gunsch,  (2004)  [14].  The  former  study  showed  that 
when  good  performance  is  overlooked  by  an  appraisal  system  in  a  job  environment, 
employees  develop  a  lack  of  trust  in  their  employer.  On  the  other  hand,  when  they  felt  that 
the  appraisal  system  was  fair,  their  trust  for  top  management  increased  and  they  regarded 
integrity  as  the  most  important  factor  for  this  trust  perception.  The  study  was  based  on  the 
standard  elements  of  trust  and  trustworthiness,  i.e.,  the  ability,  benevolence,  integrity,  and 
trustors'  propensity  as  proposed  by  Mayer  et  al  in  1995  [1].  In  the  second  example.  Biros, 
Daly,  and  Gunsch,  2004  [17]  presented  a  study  where  the  objective  was  to  see  how  people's 
usage  of  and  dependence  on  system  automation  (in  other  words  their  trust  in  system) 
changes  when  they  experience  high  task  load,  especially  under  information  uncertainty 
situations.  It  was  found  that  when  task  load  (and  hence  cognitive  load)  increases,  people 
continue  to  rely  on  the  (interaction  and  decision  support)  system,  even  if  they  have  less  trust 
in  it.  Under  high  load  and  critical  task  situations,  it  has  also  been  found  that  people  trust  the 
system  more  when  the  system  behaves  in  a  polite  manner  and  depicts  accepted  etiquette 
norms.  People  also  show  increased  trust  in  the  system  when  the  system  shows  and  maintains 
its  reliability  and  dependability. 

While  many  of  these  studies  that  did  try  to  find  a  relationship  between  trust  and  workload, 
failed  to  provide  a  detailed  account  of  the  relationship.  Namely,  the  question  of  how  various 
levels  of  cognitive  load  would  affect  the  trust  perception  or  trustworthiness  still  remains  to  be 
answered.  With  emerging  technologies  being  available  for  real-time,  automatic,  and  non- 
intrusive  measurement  of  cognitive  load,  this  could  provide  a  new  dimension  into 
understanding  trust  judgements  in  work  situations,  or  during  complex  problem  solving. 
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4.  User  Study  Design  and  Materials 

Hypotheses 

As  a  first  step  to  gain  insight  into  relationship,  we  can  pose  the  following  hypotheses 
concerning  the  interdependence  of  cognitive  load  and  trust: 

1.  For  a  fixed  level  of  trustworthiness,  increasing  the  task  complexity  (implicitly 
cognitive  load)  will  affect  both  the  likelihood  of  a  person  to  rely  more  heavily  on 
others  and  the  degree  of  trust  they  invest  in  them. 

2.  For  a  fixed  level  of  task  complexity,  varying  the  trustworthiness  of  others  will  affect 
both  the  likelihood  of  the  person  to  rely  more  heavily  on  them  and  hence  the  degree 
of  cognitive  load  they  perceive  during  the  task. 

3.  High  cognitive  load  situations  are  more  likely  to  affect  trust  judgements  that  rely  on 
accurate  assessments  of  ability  and  possibly  integrity  aspects  -  since  these  have  been 
classified  as  cognitive  rather  than  affective  processes  during  trust  judgements. 

4.  Cultural  factors  can  affect  the  interdependence  of  cognitive  load  and  trust,  such  that 
cultural  biases  in  trust  will  be  exacerbated  under  high  cognitive  load. 

Design:  Independent  Variable  and  Repeated  Measures 

To  produce  higher  cognitive  load  tasks,  a  dual-task  paradigm  was  employed.  Subjective 
ratings  of  complexity  and  difficulty  were  employed  after  each  task  set,  to  ensure  that  the 
desired  levels  of  load  built  into  the  task  design  were  actually  being  perceived  by  the  study 
participants.  The  full  set  of  all  experimental  conditions  for  a  given  cultural  factor  can  be  seen 
in  Table  1. 

Table  1.  Examination  of  interdependence  between  trust  and  cognitive  load:  experimental 
conditions. 


Trust/Trustworthiness 

Low 

High 

Cognitive  load 

Low 

Single  task,  low  induced 
trust 

Single  task,  high  induced 
trust 

High 

Dual  task,  low  induced  trust 

Dual  task,  high  induced 
trust 

In  terms  of  cognitive  load  measurement,  the  pair  of  high-trust  tasks  allows  a  'control' 
condition,  where  we  expect  to  see  variation  due  only  to  high  cognitive  load  that  matches 
previously  observed  results  (e.g.  [6]).  The  task  design  ensures  that  multiple  methods  of 
cognitive  load  measurement  are  made,  in  particular,  recordings  of  participants'  speech  and 
logged  keystrokes/mouse  movement  (behavioral  measure)  and  the  aforementioned  self¬ 
rating  (subjective  measure).  The  present  study  will  use  a  2  (cognitive  load,  w/n)  x  3  (AIB 
indicators,  w/n)  x  3  (culture,  b/w)  mixed  design. 

All  subjects  will  need  to  complete  both  load  conditions  (low  and  high  load)  in  a  repeated 
measures  design  because  the  implicit  load  measures  are  dependent  on  a  baseline  to  high  load 
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comparison.  Expected  changes  in  recorded  behavioural  data  features  of  load  will  trend  one 
way  (e.g.  increased  pauses  in  speech  during  think-aloud  of  high  load  tasks) . 

Experimental  Platform 

The  experimental  platform  simulates  a  computer-based  applicant  screening  process  called  the 
"Human  Resources  Applicant  Selection  Tool".  Participants  were  told  that  they  were 
participating  in  a  user  evaluation  of  a  new  virtual  interview  tool  being  considered  for  a 
business  application. 

Participants  assessed  potential  job  candidates  and  review  the  applicants'  virtual  resume 
which  includes  standard  experiential  data  (i.e.,  education,  previous  experience,  skills,  etc.), 
interest  statements,  and  referential  data  provided  by  previous  supervisors.  The  aim  was  that 
the  application  would  have  a  similar  look  and  feel  to  that  of  Facebook.  After  several  design 
discussions,  story  boarding  and  wireframe  iterations,  the  final  application  was  produced  and 
is  illustrated  in  Fig. 2  -18.  Candidate  applicants'  ability,  integrity,  and  benevolence  (AIB 
trustworthiness  indicators)  were  manipulated  through  referential  data  inserted  into  the  tool 
as  well  as  through  narratives  provided.  Each  applicant  was  described  by  previous 
supervisors  or  co-workers  as  being  high  or  low  one  of  the  trustworthiness  indicators. 
Examples  of  vignette-like  descriptions  of  the  trustworthiness  indicators  from  previous 
research  (see  Mayer  &  Norman,  2004)  were  adapted  for  use  in  the  current  study.  Four 
applicants  were  presented  for  each  session:  1  high  benevolence  subject,  1  high  ability 
candidate,  1  high  integrity  candidate  and  1  negative  candidate  on  all  three  aspects. 

For  the  cognitive  load  manipulation,  a  secondary  monitoring  task  was  introduced,  known  as 
the  notification  feature,  which  allowed  subjects  to  receive  and  "queue"  new  incoming 
resumes  and  applications  to  be  "processed  later".  This  was  presented  as  an  additional  feature 
of  the  tool  -  and  participants  were  asked  to  complete  two  sessions  of  the  task  set,  with  and 
without  the  notification  feature.  The  candidates  provided  in  each  round  were  different 
instances,  but  represented  the  same  AIB  manipulations  such  that  the  entire  task  was  exactly 
the  same  except  for  the  notification  feature. 

Pre  Screening  Survey  and  Training 

The  pre-screening  survey  was  implemented  using  Survey  Monkey,  over  the  web.  It  consisted 
of  13  questions  with  91  items  altogether.  The  full  Pre-Screening  survey  used  can  be  found  in 
the  Appendix  to  this  document. 

The  training  video  was  a  6-minute  animated  instruction  manual  on  how  to  complete  the 
tasks.  Some  still  shots  of  the  storyboard  of  the  video  have  been  included  in  the  Appendix 
also. 


Task  Structure 

In  each  session,  the  first  part  involved  participants  filling  out  a  mood  questionnaire  to 
ascertain  how  they  left  at  that  moment,  to  be  able  to  check  later  whether  their  current 
emotional  state  affected  their  judgment  of  the  candidates.  Participants  were  asked  to  select 
their  level  of  emotion  for  a  number  of  different  emotions,  along  a  semantic  differential  scale, 
ranging  from  "Not  at  all"  to  "Extremely",  using  radio  buttons.  As  they  were  completed,  a 
green  tick  appeared  at  the  end  of  the  row,  denoting  the  question  has  been  answered. 


-8- 


AQARD-10-4130 


Effective  of  Cognitive  Load  on  Trust 


Initial  Survey 

Using  the  following  scale,  Indicate  to  what  extent  you  feel  this  way  right  now,  that  is,  at  the  present  moment. 

Interested 

Excited 

Enthusiastic 
Alert 

Determined 
Distressed 

Upset 
Irritable 

Nervous 
Jittery 


Figure  1 :  Initial  Mood  Survey 

Immediately  following  this,  three  subtasks  were  performed  on  the  available  candidates. 


Very  Slightly  or  Not 

at  All  A  Little  Moderately  Quite  a  Bit  Extremely 


Subtask  One:  Screening  Candidates 


After  reading  a  brief  vignette  about  each  candidate  applicant,  participants  were  asked  to  rate 
their  degree  of  trust  for  each.  The  first  subtask  consisted  of  answering  a  series  of  survey 
questions  about  the  candidate  by  clicking  on  a  radio  button  on  a  semantic  differential  scale,  as 
to  whether  they  agree  or  disagree  with  a  statement  being  made  in  relation  to  this  candidate 
applicant.  One  by  one,  they  were  asked  to  review  the  candidates  under  each  of  the  four  tabs 
on  the  screen,  paying  particular  attention  to  the  comments  included  in  each  section 
(Education,  Experience  and  Volunteer  Experience  and  Personal  Interests,  as  seen  in  Fig.  2). 

The  comment  areas  were  formatted  to  stand  out  on  the  profile  page,  and  the  rest  of  the 
applicant  information  was  very  similar  for  all  candidates  (e.g.  most  had  around  the  same 
amount  of  experience  and  education).  Essentially,  the  differentiators  could  only  be  found  in 
the  comments  from  previous  employers,  supervisors  and  peers  for  that  candidate  -  this  was 
where  the  AIB  manipulations  were  made.  Additionally,  the  applicants  were  "anonymous"  - 
they  were  given  code  names  such  as  "Applicant  68K"  to  avoid  any  bias  that  may  be 
introduced  in  the  participants  upon  learning  the  candidate's  name.  Icons  were  used  instead 
of  photographs  also  to  avoid  any  bias  relating  to  the  candidate's  appearance. 
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‘  Applicant  68-K  Applicant  59 -Q  ^  ^  Applicant  24 -V 


Applicant  68-K 

Manager  at  Diaw  Lighting  and 
Configuration,  Inc. 


j  jjggj  Experience 


Education 


Jackson  University  of  Liberal  Arts,  1995 

Bachelor  of  Aits:  Business 


Diaw  Lighting  and  Configuration,  Inc.,  2001  -  Present 
Project  Manager  /  Consultant 

•  Responsibilities:  Job  duties  involved  leading  a  team  ot 
and  developing  team  cohesiveness. 


Comment  Box 


Comment  Box 


"Has  a  strong  moral  foundation.  Always  does  the  right  thing  even  in 
morally  difficult  situations."  -  Executive  Supervisor 


"Acts  on  the  up  and  up  from  what  I've  seen  and  heard. 
Always  fair  and  honest."  -  Professor  of  Business  Finance 


Boris,  Blackwood,  &  Harris,  Inc.,  1995  -  2001 
Human  Services  Manager 

•  Responsibilities:  Organization  and  development  of  product  development 


<3* 


Volunteer  Experience  and  Personal  Interests 


Weekend  Readers  Book  Club,  2001  -  Present 

Member 

•  Assisted  in  the  organization  and  planning  of  book  discussions  for  club  members. 

Comment  Box 

"Is  honest,  just,  and  impartial  without  fail."  -  Co-Club  Member 

Fhe  applicant  would  be  very  capable  of  performing 
his/her  job. 

applicant  would  be  success! 
tries  to  do. 

rhe  applicant  would  have  knowledge  about  the  work  that 
needs  done. 

I  feel  very  confident  about  the  applicant's  skills. 


Strongly  Somewhat  Neither  Agree  Somewhat 

Disagree  Disagree  Disagree  or  Disagree  Agree 

o  o  o  o  o 


Agree  Strongly  Agree 

o  o 


II 

II 


Figure  2  :  Applicant  Vignette  including  Education,  Experience  and  Personal  Interests. 


As  the  participant  completed  the  survey  questions,  a  green  tick  mark  appeared  at  the  end  of 
each  row,  signaling  to  the  participant  that  that  question  had  been  filled  in,  as  seen  in  Fig.  3. 
Once  all  the  questions  on  each  page  were  filled  in,  and  all  green  ticks  appeared  on  the  right 
hand  side  of  the  table  did  the  "Next"  button  become  activated,  to  allow  the  participant  to 
continue  moving  onto  the  next  page. 


\  ^  Agplicamt  68-K  AppIiL  .till  'j9  U  Applicant  MV  j’-  i.piQn  ant  47  S  | 


Applicant  68-K 

Manager  at  Diaw  Lighting  and 
Configuration,  Inc. 


Education 


Experience 

Diaw  Lighting  and  Configuration,  Inc.,  2001  -  Present 
Project  Manager  /  Consultant 

•  Responsibilities:  Job  duties  involved  leading  a  team  of  sales  associates 
and  developing  team  cohesiveness. 


Jackson  University  of  Liberal  Arts,  1995 

Bachelor  of  Arts:  Business 


A 


"Acts  on  the  up  and  up  from  what  I've  seen  and  heard. 
Always  fair  and  honest."  -  Professor  of  Business  Finance 


P 


"Has  a  strong  moral  foundation.  Always  does  the  right  thing  even  in 
morally  difficult  situations."  -  Executive  Supervisor 


Boris,  Blackwood.  &  Harris,  Inc.,  1995  -  2001 
Human  Services  Manager 

•  Responsibilities:  Organization  and  development  of  product  development 


<3* 


Volunteer  Experience  and  Personal  Interests 


Weekend  Readers  Book  Club,  2001  -  Present 
Member 

•  Assisted  in  the  organization  and  planning  of  book  discussions  for  club  members. 
~p)  "Is  honest,  just,  and  impartial  without  fail."  -  Co-Club  Member 


Figure  3:  Green  ticks  showing  progress  through  the  survey  screening  questions 
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Subtask  Two:  Filling  Candidates  into  Positions 


Once  the  subject  completed  the  review  of  all  four  applicants  individually,  they  were  asked  to 
bin  the  applicants  in  three  categories:  1)  decision/action  oriented  -  whom  the  participants 
would  most  trust  as  their  boss,  2)  relationship  oriented  -  whom  the  participants  would  most 
trust  as  their  friend/colleague,  and  3)  judgment  oriented  -  whom  the  participants  would  most 
trust  to  be  someone  else's  supervisor  (but  not  the  participant  themselves). 


j  9 Fill  Positions  [  1  : 


^  Applicant  68-K 

^  Applicant.  59-Q 

''‘"Acts  on  the  up  and  up  from  what  I've  seen  and  heard. 
Always  fair  and  honest  ".  Professor  of  Business  Finance 

'''"Has  a  strong  moral  foundation.  Always  does  the  right 
thing  even  in  morally  difficult  situations.",  Executive 

Supervisor 

•''"Is  honest,  just,  and  impartial  without  fail.",  Co-Club 
Member 

A* 11  Is  kind  and  caring  toward  others.  Goes  well  above  and 
beyond  the  call  of  duty  to  look  out  for  other  students'  best 
interests.".  Professor  of  Business  Finance 

'-'"Puts  forth  an  extra  effort  to  support  others.  Takes  good 
care  of  the  employees.".  Executive  Supervisor 

'-'"Bends  over  backwards  to  help  you  out  whenever  possible. 
Has  always  been  particularly  good  to  me.",  Co-Club  Member 

^  Applicant  24-V 

^  Applicant  47-S 

'''"Judgment  is  not  always  ethically  sound.",  Professor  of 
Business 

'''"Does  not  have  a  strong  set  of  principles.  Lacks  a  moral 
center ",  Executive  Supervisor 

'''"We  have  known  each  other  for  about  two  years  ", 

Co- Club  Member 

'''"Consistently  demonstrated  strong  skills  on  class  projects 
and  understanding  of  course  material.".  Professor  of 
Management 

'-'"A  good  employee  who's  technical  know-how  is  widely 
regarded  as  state-of-the-art.".  Executive  Supervisor 

^"Proficient  and  competent  while  completing  work. 

Performs  quality  work.  Always  gets  things  done  well.",  Co- 

Club  Member 

Drag  and  Drop  the 
applicant  to  a  position. 

Please  select  the  level  of 
confidence  you  have  in 
your  decision. 


Your  Supervisor : 

I  .  Drop  Applicant  Here 


Unselected 


Drop  Applicant  Here 


Other's  Supervisor: 

Vj.  Drop  Applicant  Here 


|lfnseli 


Figure  4:  Fill  Positions  for  Supervisor,  Co-worked  and  Other's  Supervisor 


Figure  4  shows  the  second  subtask,  filling  in  the  positions  of  supervisor,  co-worker  and 
colleague  from  the  available  candidates.  In  order  to  facilitate  this,  summaries  of  the 
candidates  profiles  were  presented  on  a  single  page,  each  showing  the  information  in  the 
"comment"  areas  of  the  individual  profiles  in  the  previous  subtask.  The  profiles  were 
presented  in  the  same  order  as  the  tabs  in  the  previous  subtask,  but  arranged  on  a  grid  from 
top  to  bottom  and  left  to  right. 

The  candidate  profiles  could  be  dragged  to  the  positions  shown  on  the  right  hand  column  of 
the  window,  as  the  participant  chooses.  Once  the  candidate  is  dragged  to  a  position,  the 
profile  summary  is  greyed  out  to  indicate  that  profile  cannot  be  chosen  for  any  other 
position.  The  participants  were  also  required  to  fill  in  some  information  about  how  confident 
they  were  that  they  were  choosing  the  right  person  for  the  position,  along  a  semantic 
differential  scale  of  5  points  using  a  drop  down  box,  from  "Extremely  confident"  to  "Very 
little  confidence"  in  their  choice  for  this  position. 
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|  3  Fill  Positions  j  jj  Rank  applicants 


J  Applicant  68-K 


'-'"Acts  on  the  up  and  up  from  what  I’ve  seen  and  heard. 
Always  fair  and  honest.".  Professor  of  Business  Finance 

'-'"Has  a  strong  moral  foundation.  Always  does  the  right 
thing  even  in  morally  difficult  situations. ",  Executive 
Supervisor 

^"Is  honest,  just,  and  impartial  without  fail.",  Co- Club 


^  Applicant  24-V 


'''"Judgment  is  not  always  ethically  sound. ",  Professor  of 
Business 

''’"Does  not  have  a  strong  set  of  principles.  Lacks  a  moral 
center  ".  Executive  Supervisor 

A  "We  have  known  each  other  for  about  two  years.", 

Co- Club  Member 


Applicant  59-Q  (Supervisor) 


^  Applicant  47-S 


'''"Consistently  demonstrated  strong  skills  on  class  projects 
and  understanding  of  course  material.".  Professor  of 
Management 

'-'"A  good  employee  who’s  technical  know-how  is  widely 
regarded  as  state-of-the-art.",  Executive  Supervisor 

'-'"Proficient  and  competent  while  completing  work. 
Performs  quality  work.  Always  gets  things  done  well.",  Co- 

Club  Member 


Drag  and  Drop  the 
applicant  to  a  position. 

Please  select  the  level  of 
confidence  you  have  in 
your  decision. 


Your  Supervisor : 

1 1 


Applicant  59-Q 


Ho** 


ate  Confidence 


Co-Worker : 


,  Drop  Applicant  Here 


Other  1 s  Supervisor : 


i  Drop  Applicant  Here 


Figure  5  Filling  in  a  candidate  for  a  Supervisor  position,  with  moderate  confidence. 


Once  all  positions  are  filled  (since  there  are  three  positions,  only  three  out  of  four  candidates 
can  be  selected),  the  continue  button  is  activated,  as  seen  in  Fig.  6  and  the  participant  is  then 
asked  to  type  in  a  justification  for  their  choice.  A  pop  up  dialogue  box  is  provided  for  this 
purpose,  and  allows  the  participants  to  explain  their  reasons  for  choosing  each  candidate  for 
each  role,  and  why  they  judged  them  to  be  the  best  fit.  Once  their  reasons  have  been  typed  in, 
they  can  hit  the  Save  button  on  the  dialogue  box  and  move  on  to  the  next  subtask.  This  is 
illustrated  in  Fig.  7. 

j  at  Fill  Positions  [ 


^  Applicant  68-K 

'-'"Acts  on  the  up  and  up  from  what  I've  seen  and  heard. 
{Always  fair  and  honest.".  Professor  of  Business  Finance 

'-'"Has  a  strong  moral  foundation.  Always  does  the  right 
thing  even  in  morally  difficult  situations.".  Executive 
Supervisor 

'■'"Is  honest,  just,  and  impartial  without  fail.",  Co- Club 
Member 


,  Applicant  24-V  (Other's  Supervisor) 


,  Applicant  59-Q  (Supervisor) 


Applicant  47-S  (Co-Worker) 


Drag  and  Drop  the 
applicant  to  a  position. 

Please  select  the  level  of 
confidence  you  have  in 
your  decision. 


Your  Supervisor : 
L&  Applicant  59-Q 


jllodei 


(rate  Confidence 


Co-Worker : 


Applicant  47-S 


jvery  High 


Confidence 


Other's  Sigiervisor: 


Applicant  24-V 


Some  Confidence 


Figure  6  :  All  positions  filled.  Continue  Button  is  activated. 
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j  28  Fill  Positions  j  l  Rank  Applicants  | 


^  Applicant  68-K 

•''"Acts  on  the  up  and  up  from  what  I've  seen  and  heard. 
Always  fair  and  honest. ",  Professor  of  Business  Finance 


Applicant 


59-Q 


(Supervisor) 


Drag  and  Drop  the 
applicant  to  a  position. 


'-'"Has  a  strong  moral  foundation.  Always  does  the  right 
thing  even  in  morally  difficult  situations.".  Executive 
Supervisor 


9 


Please  select  the  level  of 
confidence  you  have  in 
your  decision. 


r*  Is  honest.  Just,  and  impartial  without  fail ",  Co-Club 


,  Applicant-  24-V  (Other's  Supervisor) 


Applicant  47-S  (Co-Worker) 


Your  Supervisor: 


Continue 


Figure  7:  Justification  Pop-Up  Dialogue  Box 


Subtask  Three  :  Ranking  Candidates 

The  last  subtask  involves  ranking  the  candidates  in  terms  of  suitability  for  each  position.  This 
means  that  all  four  candidates  are  placed  in  order  from  most  suitable  to  least  suitable  for  each 
position.  The  rankings  are  made  for  Supervisor,  Co-worker  and  Other's  Supervisor  in  that 
order,  as  can  be  seen  in  Fig.  8,  10  and  11.  Once  all  candidate  profiles  have  been  ranked,  the 
"Continue"  button  is  activated  and  the  participant  can  move  on  from  this  subtask. 


f  a  Fill  Positions  j  Rank  Applicants 


J  Applicant  68-K 


'-'"Acts  on  the  up  and  up  from  what  I've  seen  and  heard. 
Always  fair  and  honest  ".  Professor  of  Business  Finance 

A1  "Has  a  strong  moral  foundation.  Always  does  the  right 
thing  even  in  morally  difficult  situations.".  Executive 
Supervisor 

'-'"Is  honest,  just,  and  impartial  without  fail.",  Co-Club 
Member 


J  Applicant  24-V 


-•'"Judgment  is  not  always  ethically  sound.".  Professor  of 
Business 

'-’"Does  not  have  a  strong  set  of  principles.  Lacks  a  moral 
c  enter ",  Executive  Supervisor 

'-’"We  have  known  each  other  for  about  two  years. ", 

Co- Club  Member 


^  Applicant  59-Q 


'-'"Is  kind  and  caring  toward  others.  Goes  well  above  and 
beyond  the  call  of  duty  to  look  out  for  other  students'  best 
interests  ".  Professor  of  Business  Finance 


'-'"Puts  forth  an  extra  effort  to  support  others.  Takes  gi  d 
care  of  the  employees.".  Executive  Supervisor 


r* "Bends  over  backwards  to  help  you  out  whenever  pos  ble. 
Has  always  been  particularly  good  to  me.1',  Co-Club  Me 


^  Applicant  47-S 


'-’"Consistently  demonstrated  strong  skills  on  class  projects 
and  understanding  of  course  material,".  Professor  of 
Management 

'-’"A  good  employee  who's  technical  know-how  is  widely 
regarded  as  state-of-the-art.".  Executive  Supervisor 

'-'"Proficient  and  competent  while  completing  work. 
Performs  quality  work.  Always  gets  things  done  well.",  Co- 

Club  Member 


Please  rank  all  applicant 
from  best  (1)  to  worst  (4) 
choice  for  each  position. 


Your  Supervisor: 

(1)  Drop  Applicant  Here 


.  (2)  Drop  Applicant  Here 

.  (3)  Drop  Applicant  Here 
(4)  Drop  Applicant  Here 


Figure  8  :  Ranking  for  positions  -  Supervisor 
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(  Fill  Positions  1  -g  Rank  Applicants 


applicant  68-K  (2nd) 


^  Applicant  24-V 


'''"Judgment  is  not  always  ethically  sound.".  Professor  of 
Business 

'''"Does  not  have  a  strong  set  of  principles.  Lacks  a  moral 
center  ".  Executive  Supervisor 

'''"We  have  known  each  other  for  about  two  years.", 

Co- Club  Member 


Applicant  59-Q  (1st) 


^  Applicant  47-S 


'''"Consistently  demonstrated  strong  skills  on  class  projects 
and  understanding  of  course  material  ",  Professor  of 
Management 

A  "A  good  employee  who's  technical  know-how  is  widely 
regarded  as  state-of-the-art.".  Executive  Supervisor 

'''"Proficient  and  competent  while  completing  work. 
Performs  quality  work.  Always  gets  things  done  well.",  Co- 

Club  Member 


Please  rank  all  applicant 
from  best  (1)  to  worst  (4) 
choice  for  each  position. 


Your  Supervisor: 


(1)  Applicant  59-Q 

(2)  Applicant  68-K 

k  (3)  Drop  Applicant  Here 
(  (4)  Drop  Applicant  Here 


Figure  9  :  Ranking  for  Positions  -  Supervisor,  with  initial  selections  filled 


^  Applicant  68-K 

%  Applicant  59-Q 

'''"Acts  on  the  up  and  up  from  what  I've  seen  and  heard. 
Always  fair  and  honest  ",  Professor  of  Business  Finance 

'''"Has  a  strong  moral  foundation.  Always  does  the  right 
thing  even  in  morally  difficult  situations.".  Executive 

Supervisor 

'''"Is  honest,  just,  and  impartial  without  fail.",  Co-Club 
Member 

>Is  kind  and  caring  toward  others.  Goes  well  above  and 
beyond  the  call  of  duty  to  look  out  for  other  students'  best 
interests.".  Professor  of  Business  Finance 

'''"Puts  forth  an  extra  effort  to  support  others.  Takes  good 
care  of  the  employees  ".  Executive  Supervisor 

'''"Bends  over  backwards  to  help  you  out  whenever  possib 
Has  always  been  particularly  good  to  me.",  Co-Club  Memb 

?r 

J  Applicant  24-V 

^  Applicant  47-S 

'•'’"Judgment  is  not  always  ethically  sound.".  Professor  of 
Business 

'''"Does  not  have  a  strong  set  of  principles.  Lacks  a  moral 
center  ".  Executive  Supervisor 

''’"We  have  known  each  other  for  about  two  years. ", 

Co- Club  Member 

'''"Consistently  demonstrated  strong  skills  on  class  projects 
and  understanding  of  course  material  ",  Professor  of 
Management 

'''"A  good  employee  who's  technical  know-how  is  widely 
regarded  as  state-of-the-art.".  Executive  Supervisor 

'-'"Proficient  and  competent  while  completing  work. 

Performs  quality  work.  Always  gets  things  done  well.",  Co- 

Club  Member 

Please  rank  all  applicant 
from  best  (1)  to  worst  (4) 
choice  tor  each  position. 


Co-Worker 


ttaa  maun  H— 


(2)  Drop  Applicant  Here 
k  (3)  Drop  Applicant  Here 
(4)  Drop  Applicant  Here 


Figure  10  :  Ranking  for  the  Co-worker/Colleague  position 
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^  Applicant  68-K 

^  Applicant  59-Q 

•-'"Acts  on  the  up  and  up  from  what  I've  seen  and  heard. 
Always  fair  and  honest  ".  Professor  of  Business  Finance 

•-'"Has  a  strong  moral  foundation.  Always  does  the  right 
thing  even  in  morally  difficult  situations  ".  Executive 
Supervisor 

'-"is  honest,  just,  and  impartial  without  fail.",  Co- Club 
Member 

?'ls  kind  and  caring  toward  others.  Goes  well  above  and 
beyond  the  call  of  duty  to  look  out  for  other  students’  best 
interests  ".  Professor  of  Business  Finance 

'-'"Puts  forth  an  extra  effort  to  support  others.  Takes  good 
care  of  the  employees  ".  Executive  Supervisor 

'-’ "Bends  over  backwards  to  help  you  out  whenever  possible 

^  Applicant  24-V 

^  Applicant  47-S 

•-"'Judgment  is  not  always  ethically  sound.".  Professor  of 
Business 

'-'"Does  not  have  a  strong  set  of  principles.  Lacks  a  moral 
center.".  Executive  Supervisor 

•-'"We  have  known  each  other  for  about  two  years.", 

Co- Club  Member 

^"Consistently  demonstrated  strong  skills  on  class  projects 
and  understanding  of  course  material.",  Professor  of 
Management 

'-'“A  good  employee  who's  technical  know-how  is  widely 
regarded  as  state-of-the-art.".  Executive  Supervisor 

•-'"Proficient  and  competent  while  completing  work. 

Performs  quality  work.  Always  gets  things  done  well.",  Co- 

Club  Member 

Please  rank  all  applicant 
from  best  (1)  to  worst  (4) 
choice  for  each  position. 


Other’ s  Supervisor 

(1)  Drop  Applicant  Here 


.  (3)  Drop  Applicant  Here 
.  (4)  Drop  Applicant  Here 


Figure  11 :  Ranking  for  the  Other's  Supervisor  position 


Task  Review 

The  final  section  is  a  Task  Review,  where  some  metrics  are  provided  about  the  number  of 
applicants  that  have  been  rated,  selected  for  positions  and  ranked  in  the  above  tasks.  The 
"Future  Tasks  Queued"  section  is  only  used  in  the  high  load  'round',  where  the  Notification 
feature  is  activated.  The  participants  final  score  will  be  shown  here,  namely,  the  number  of 
"Human  Resources"  notifications  were  spotted  and  added  to  the  queue. 


(  M  Fill  Position-;  Rank  Applicants 


Task  Review: 


Thank  you  for  ranking 
the  applicants. 


Rating  Applicants 

Number  of  Applicants  that  were  rated:  4  of  4  (100%) 

Position  Selection 

Number  of  positions  filled:  3  of  3  (100%) 


Please  click  continue 
when  you  are  ready  to 
take  some  surveys. 


Ranking 

Number  of  Applicants  Ranked:  4  of  4  (100%) 
Number  of  Positions  Ranked:  3  of  3  (100%) 


Future  Tasks  Queued  (Not  Used  In  This  Session) 


Applicants  added  for  Human  Resource 
Manager: 


Not  Used  In  This  Session 


Incorrectly  added  applicants:  Not  Used  In  This  Session 


Figure  12  :  The  Task  Review  window 


At  the  end  of  each  round,  the  participants  were  asked  to  rate  on  a  Likert  scale,  with  a  radio 
button,  how  difficult  they  found  the  session.  We  expected  that  they  would  rank  the  session 
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with  the  Notification  feature  to  be  relatively  more  difficult  than  the  session  without  the 
Notification  feature. 


Post-Session  Survey 

Using  the  scale  provided,  please  rate  your  agreement  with  the  following  item. 


Overall,  how  difficult  did  you  find  this  task? 


Extremely  easy 

Easy 

Slightly  Easy 

Neither  easy 
nor  difficult 

Slightly  difficult 

Extremely 

difficult 

o 

o 

o 

o 

o 

o 

Figure  13  :  Subjective  Difficulty  Likert  Scale 


Fifty-percent  of  subjects  completed  the  low-load  session  first,  while  the  rest  completed  the 
high  load  session  first.  An  instruction  appeared  before  the  participant  started  the  high  load 
session,  reminding  them  of  how  they  were  expected  to  handle  incoming  notifications,  as  in 
Fig.  12..  This  was  a  single  snapshot  recap  of  the  training  video,  where  the  whole  procedure 
was  shown  of  spotting  and  queuing  incoming  notifications  was  explained. 


This  session  contains 
the  pop-up  feature! 

•  Click  and  add  pop  ups  for 
"Human  Resource 
Manager" 

—  Disregard  all  other 
positions 


•  Border  can  turn  from 
Green  to  Red 
-  Indicates  performance 


Figure  14:  Instruction  indicating  that  the  Notification  feature  would  be  used  in  this 

Session. 

The  subtasks  to  be  completed  in  notifications  would  appear  periodically,  for  5  seconds  at  a 
time,  at  the  bottom  right  hand  corner  of  the  screen  during  the  entire  session.  The  following 
format  is  used: 

Applicant  28 -U  (MED) 

for 

Human  Resources 

Figure  15:  Notification  Item 
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The  incoming  notifications  would  be  labeled  as  belonging  to  different  departments,  such  as 
Recruiter,  Finance  or  Human  Resources  for  example,  as  can  be  seen  in  Fig.  15  and  Fig.  16.  The 
participants  were  asked  to  ignore  all  notifications,  except  those  addressed  for  "Human 
Resources".  The  participant  has  4  seconds  to  click  on  the  notification  item,  which  would  bring 
up  a  task  list,  to  be  populated  with  "Human  Resources"  notifications.  Once  selected,  the  item 
could  then  be  added  to  this  list  at  the  end  of  the  queue.  Participants  could  also  choose  to 
refrain  from  adding  the  item  onto  the  list  by  simply  closing  the  window. 

If  the  participant  missed  adding  "Human  Resources"  notifications,  or  incorrectly  added 
notifications  from  other  departments,  the  border  of  the  notifications  would  change  from 
green  to  amber  to  red,  to  indicate  that  the  participants  should  pay  more  attention  to  the 
notifications.  When  a  sufficient  number  of  "Human  Resources"  notifications  are  added  to  the 
list  again,  the  borders  on  new  notifications  will  appear  green  once  again. 

The  number  of  correct  additions  overall  is  tallied  throughout  the  session  and  a  final  score  is 
given  at  the  end,  in  the  Task  Review. 


Figure  16  :  A  Notification  appearing  for  a  Recruiter. 
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applicant  35-U  \  X  Applicant  65-W  f  £  Applicant  81-D  ^  ^  Applicant  76-0 


Applicant  35-U 


m 


Experience 


Project  Manager  at  Distal,  Inc. 


Education 


Northern  University,  1998 

Bachelor  of  Arts :  Business 


"Consistently  demonstrated  strong  skills  on  class  projects 
-=)  and  understanding  of  course  material. "  -  Professor  of 

Business 


Distal,  Inc.,  2003  -  Present 

Project  Manager 

•  Responsibilities:  Supervised  employees  on  product  development. 

Developed  a  model  for  employees  to  follow  to  increase  performance  and 
labor  costs. 

"Is  a  valuable  employee  who  performed  at  a  high  level.  "  -Executive 
Supervisor 


Arrow  Consulting,  1998-  2003 
Manager 

•  Responsibilities:  Led  com 
Met  deadlines  and  provid 


<0* 


Volunteer  Experience  and  Personal  Interests 


The  Organization  for  Creative  Writers,  2003  -  Present 
Member 

•  Assisted  in  development  of  an  online  writing  portal  for  group  members  to  share  material  and  products 
)  "Performs  Quality  work"  -  Co-Club  Member 


Task  List 

Task  List 

HI 

Applicant 

Priority 

The  applicant  would  be  very  capable  of  performing 
his/her  job. 

applicant  would  be  successful  at  j 

tries  to  do. 


Strongly  Somewhat  Neither  Agr 

Disagree  Disagree  Disagree  or  Disagre 

o  o  o  o 


The  applicant  would  have  knowledge  about:  rh*  um-i;  rhar - 

needs  done. 

feei  very  confident  etout  the  applied  "Human  Resources"  Notification 


Applicant  35-0  (HIGH) 
for 

Human  Resource  Manager 


Figure  17  :  Adding  a  Notification  for  Human  Resources  to  the  Task  List  (queue) 


.  Applicant  35-U  |  Applicant  65-W  Applicant  81-D  ^ 


£ 

W 


Applicant  35-U 


m 


Experience 


Project  Manager  at  Distal,  Inc. 


Education 


Northern  University,  1998 

Bachelor  of  Arts:  Business 


"Consistently  demonstrated  strong  skills  on  class  projects 
±)  and  understanding  of  course  material. "  -  Professor  of 
'  Business 


Distal,  Inc.,  2003  -  Present 
Project  Manager 

•  Responsibilities:  Supervised  employees  on  product  development. 
Developed  a  model  for  employees  to  follow  to  increase  performance  and 
labor  costs. 

"Is  a  valuable  employee  who  performed  at  a  high  level.  "  -Executive 
^  Supervisor 

Arrow  Consulting,  1998-  2003 
Manager 

•  Responsibilities:  Led  consulting  projects  on  various  business  issues. 

Met  deadlines  and  provided  quality  products  to  customers. 


Volunteer  Experience  and  Personal  Interests 


The  Organization  for  Creative  Writers,  2003  -  Present 
Member 

•  Assisted  in  development  of  an  online  writing  portal  for  group  members  to  share  material  and  prod 
i  "Performs  Quality  work"  -  Co-Club  Member 


Applicant  35-0 


The  applicant  would  be  very  capable  of  performing 
his/her  job. 

applicant  would  be  successful  at 

tries  to  do. 


Strongly 

sagre 

o 


Somewhat  Neither  Agret 
Disagree  Disagree  or  Disagree 

o  o  o 


things  he /she 


Applicant 


Priority 


The  applicant  would  have  knowledge  about  the  work  that 
leeds  done. 

feel  very  c 


o 


” _ 

|  O  A.I.I 


Strongly  Agree 

o  v| 


S! 

i 


Applicant  97-E  (HIGH) 
for 

Human  Resource  Manager 


Figure  18  :  The  Notification  Item  has  been  added  on  to  the  Task  List  (queue) 
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Figure  19  :  Notification  item  in  the  Ranking  subtask 
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Modalities  and  Data  Streams 

A  number  of  modalities  and  data  streams  have  been  collected  in  this  experiment.  The 
Experimental  Platform  described  in  the  previous  section  was  developed  in-house,  that 
incorporates  all  data  collection,  in  both  versions  (high  CL  and  low  CL). 

1.  Survey  Responses 

•  Pre-Screening  Survey 

A  pre-screening  survey  consisting  of  13  questions,  with  a  total  of  91  multiple 
choice  questions  about  the  participant's  attitudes  towards  their  supervisors  and 
peers,  honesty,  kindness  and  trustworthiness,  as  well  as  some  self-identifying 
ethnicity  and  personality  based  questions. 

•  Mood  Survey 

This  single  question  survey  required  participants  to  rate  a  series  of  affective 
aspects,  such  as  happiness  and  sadness,  according  to  how  intensely  the  feeling 
was  being  experienced  at  the  time. 

•  Subjective  ratings  of  mental  effort/  task  difficulty 

This  single  question  survey  asked  participants  to  rate  how  difficult  the  tasks 
were.  It  was  administered  at  the  end  of  both  the  high  load  and  low  load  sessions. 

2.  Behavioural  Measures 

•  Speech:  think-aloud  protocols 

Participants  were  asked  to  verbalize  their  thought  processes  as  they  work 
through  the  three  subtasks.  These  utterances  were  recorded. 

•  Justification  Text 

Typing  behavior  of  justification  for  filling  positions,  the  text  provided  will  be 
analyzed  for  temporal  and  linguistic  elements. 

•  Mouse  trajectories 

These  are  in  the  form  of  (x,y)  coordinates,  and  are  sampled  with  enough 
resolution  to  reproduce  the  entire  experiment  session.  The  trajectory  data  will  be 
used  to  track  widget  manipulation  and  log  use  of  the  mouse  as  a  placeholder  or 
pointer  by  hovering  over  specific  areas  of  the  application  window.  They  can  also 
provide  an  indication  of  attentional  focus. 

•  Other  interactive  behaviors 

Application  level  behaviors  such  as  false  starts  in  answer  selections,  changes  in 
selections,  etc  have  been  collected  and  will  be  analyzed. 

3.  Performance  Measures 

•  Ratings,  Filling  positions  and  Rankings: 

The  final  responses  to  the  actual  subtasks. 

•  Time-to-completion 

Overall  and  per  task. 

•  Performance  on  secondary  task:  Number  of  notifications  correctly  added,  time- 
to  respond,  erroneously  added  notifications  items,  errors  avoided  before  adding 
erroneous  items. 
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Procedure 

The  consent  forms  were  presented  to  each  participant  (see  Appendix:  Consent  Forms)  a  few 
weeks  before  the  study  took  place.  Participants  were  assigned  randomly  to  complete  a 
specific  experimental  condition  (high  or  low  cognitive  load)  first.  In  the  Australian  data 
collection  procedure,  due  to  time  constraints,  participants  would  go  on  to  complete  a  brief 
background  survey  online  (see  Appendix:  Pre  Screening  Survey),  in  the  week  following  the 
experiment.  However,  at  the  other  sites  (Malaysia  and  Australia)  the  students  will  complete 
this  background  survey  immediately  before  the  study  begins. 

The  experimenter  instructed  each  participant  to  sit  at  a  specific  computer  station  and  to  watch 
the  training  video,  which  explained  the  purpose  of  the  tool  (the  script  can  be  found  in  the 
Appendix).  The  training  content  will  be  couched  as  the  examination  of  a  new  applicant 
screening  HR  tool  used  to  rate  job  applicants.  An  experimenter  or  research  assistant  was  be 
present  throughout  the  entire  training  and  participants  could  ask  questions  at  any  time.  No 
practice  trial  was  provided  since  use  of  the  system  was  demonstrated  in  the  video. 

Once  the  participants  were  ready  to  begin,  they  started  the  application  and  submitted  their 
student  identifier  to  begin.  At  this  point,  they  completed  the  mood  survey.  Depending  on 
which  session  (high  or  low  load)  they  were  completing  first,  the  Notification  feature 
instruction  screen  appeared  to  remind  them  that  they  need  to  queue  notifications.  The 
participants  were  then  presented  with  the  first  subtask:  screening  applicants.  Additional 
surveys  will  be  administered  throughout  the  task  and  at  the  end  of  the  task  . 

Due  to  time  constraints,  participants  were  debriefed  the  following  week,  during  their  tutorial. 
To  further  the  participants'  learning,  an  exercise  was  provided  for  them  in  relation  to 
interface  evaluation  and  methodology  of  experimentation,  which  were  topics  covered  in  their 
course  (see  Appendix:  Debriefing  and  Exercise). 


5.  Operational  Processes 

IRB  Approvals 

All  Australian  team  members  (Fang  Chen,  Natalie  Ruiz  and  Asif  Khawaja)  have  completed 
CITI  training  and  have  received  their  certificates.  Refersher  courses  must  be  completed  every 
year  and  a  new  round  will  be  required  in  December. 

NICTA  applied  for  Federal  Wide  Assurance  (FWA)  and  received  word  of  approval  in  July 
2010  after  a  process  taking  several  months.  NICTA  also  completed  the  DoD  Addendum 
following  this  approval,  where  NICTA  formally  acknowledged  the  requirements  to  follow 
IRB  directives.  An  IAIR  was  also  needed  from  which  indicated  that  UNSW  Ethics  Committee 
would  oversee  NICTA  research  at  the  Australian  site.  In  fact,  NICTA's  local  data  collection 
and  user  research  approval  had  already  been  granted  by  UNSW  Ethics  Committee  (Ref: 
08/2010/42)  in  July  2010  and  will  last  until  September  2013.  This  approval  has  also  been 
ratified  by  the  University  of  Sydney  Ethics  Committee  (Protocol:  14031)  in  July  2011.  All 
documentation 

Finally,  cover  letters  and  IRB  package  for  "Trust  Management  Studies"  were  submitted  to  the 
AFRL  IRB  including  the  User  Study  design  and  supporting  documentation  and  approved  on 
(Approval  11-21)  the  11th  June  2011. 
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Pilots  Surveys 

The  cognitive  load  manipulations  to  be  used  will  be  determined  through  pilot  testing  for 
feasibility  with  the  overall  study  design.  Pilot  studies  were  conducted  to  ensure  that  the 
manipulations  are  effective  and  that  all  other  applicant  information  is  considered  equivalent 
across  load  levels  and  across  cultures.  Since  two  rounds  of  the  experiment  would  need  to  be 
completed  -  the  amount  of  stimulus  material  needed  to  be  doubled  also.  Appropriate 
language  will  need  to  be  tested  for  each  site  to  test  cultural  fit.  To  address  these  concerns  the 
following  pilot  surveys  were  planned  and  conducted: 

Cultural  appropriateness  of  language  and  bias  of  trust  manipulations:  One  pilot  planned  at 
each  site  (Malaysia,  Australia  and  US)  of  50  subjects  of  a  similar  background  to  the  test  group 
(university  students).  This  content  pilot,  in  the  form  of  a  web  survey,  was  deployed  through 
Survey  Monkey  (can  be  found  in  the  Appendix).  The  researchers  for  each  site  carefully 
modified  the  content  to  ensure  that  no  bias  would  be  present  in  or  between  the  candidate 
profiles  that  would  allow  differentiation  other  than  through  the  AIB  indicators  -  for  example, 
at  the  Australian  and  Malaysian  sites,  the  word  "college"  is  seen  as  less  prestigious  than  the 
word  "university"  while  in  the  US,  the  words  are  interchangeable.  Similarly,  some  sentences 
that  were  to  be  used  as  neutral  comments  could  be  interpreted  as  negative  comments  in 
Australian  culture.  For  example,  if  a  recommendation  for  a  candidate  reads  "I  have  known 
this  person  for  2  years",  the  implication  is  that  there  is  nothing  positive  to  say  about  the 
person  except  the  length  of  time  they  have  known  each  other.  Other  idioms  such  as  use  of 
"above  and  beyond"  and  "state  of  the  art"  could  cause  comprehension  issues  outside  of  the 
US. 

The  US  site  collected  60  responses  to  their  version  of  the  pilot  survey,  and  the  Australian  site 
managed  to  collect  the  responses  of  30  participants  from  the  University  of  NSW  Human 
Computer  Interaction  students.  The  Malaysian  site  has  yet  to  administer  their  content  survey 
at  the  time  of  writing. 

Pilot  Experiments 

Various  sets  of  pilot  experiments  were  also  undertaken  in  Sydney  and  in  Dayton,  with 
research  assistants.  This  helped  to  evolve  the  experimental  software,  as  well  as  the  UI  design 
and  calibrate  the  secondary  task  speed  and  specifications. 

It  was  found  that  the  "think-aloud"  protocol  added  an  element  of  uncertainty  in  terms  of  time 
-  as  it  was  found  participants  would  vary  in  the  amount  of  speech  and  the  depth  of 
explanation  they  provided.  Some  participants  finished  each  session  in  20-25  minutes,  while 
others  took  almost  90  minutes.  Additionally,  during  the  pilot  experiments,  it  was  decided  to 
reduce  the  number  of  candidates  used  in  the  experiment  session  from  6  to  4.  This  reduction 
allowed  some  regulation  of  the  timing  of  the  study,  preventing  subjects  from  taking  too  long 
and  being  exhausted  at  the  end  of  the  session.  We  also  provided  an  extra  incentive  to  the 
participants  of  a  prize  for  the  participant  in  every  session  who  was  able  to  most  consistently 
speak  throughout  the  session. 

Participants  and  Consent  Forms 

A  total  of  120  subjects  were  planned  for  the  user  study  per  site.  The  study  duration  will  be 
approximately  1  hour.  Including  pilot  studies,  a  total  of  360  subjects  will  be  recruited  to 
participate  across  the  three  study  locations.  Participants  can  either  volunteer  to  participate 
(e.g.,  military  personnel),  receive  course  credit  at  the  respective  university,  and/or  receive 
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remuneration  in  the  amount  of  $20  (or  local  equivalent)  for  each  60-minutes  of  participation, 
which  is  the  estimated  duration  of  the  study. 

In  the  Australian  study,  100  students  from  the  INF03315  course  (Human  Computer 
Interaction)  at  Sydney  University,  agreed  to  participate.  The  consent  forms  are  attached  in  the 
Appendix  of  this  document.  Of  these,  97  completed  all  sections  of  the  study.  The  students 
received  course  credit  in  exchange  for  their  participation,  as  well  as  snacks  and  movie  ticket 
prizes  for  the  "top  performers"  after  the  session. 

To  maintain  confidentiality,  we  will  use  student  identifiers  (numbers  and  letters)  when 
dealing  with  the  data.  Each  participant  will  be  assigned  a  subject  identifier  number.  The 
number  will  not  be  attached  to  any  document  that  includes  personal  identifiers  (such  as  the 
informed  consent  form).  We  will  only  request  demographic  data  such  as  (age,  sex,  class  rank, 
ethnicity).  Only  group  data  will  be  analyzed  and  all  consent  forms  and  data  will  be  stored 
under  lock  and  key  when  not  in  use  at  each  site. 


6.  Data  Collection  Summary:  Australia 

Schedule 

The  study  schedule  for  Australia,  Mayalsia  and  US  has  evolved  as  follows: 

•  The  Australian  group:  -100  students  from  the  University  of  Sydney  participated  in 
the  user  study  on  the  18th  October. 

•  The  US  group:  -160  students  will  participate  in  January  2012 

•  The  Malaysian  group:  Date  to  be  scheduled. 

The  Australian  group's  data  will  undergo  preliminary  analysis  to  determine  whether  changes 
need  to  be  made  with  the  protocol.  If  the  new  version  of  the  tool  is  substantially  different 
from  the  version  administered  in  this  round  due  to  confounds  or  other  issues,  a  new  set  of 
students  can  be  canvassed  from  Sydney  University  next  year  to  complete  the  new  version  of 
the  user  study. 

Australian  data  collection  brief  stats: 

•  91  subjects  completed  both  conditions  (high  and  low  CL) 

•  Approximately  239  survey/response  data  points  per  subject 

•  Speech  data:  6.5Gb  =  58  hours  of  speech 

•  Interactive  Behaviour:  -96  million  data  points  including  mouse  trajectories,  selection, 
typing,  browsing  activity  (attentional  focus) 
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Analysis  Plan 

The  hypotheses  described  in  the  earlier  section  of  the  same  title  have  been  operationalized  as 
follows: 


HI: 

Participants  from  a  collectivistic  culture  (e.g. Malaysia)  will  rate  trust  higher  when 
applicants  have  higher  benevolence 

H2: 

Participants  from  an  individualistic  culture  (e.g. US,  Australia)  will  rate  trust 
higher  when  applicants  have  higher  ability 

H3: 

Participants  will  bin  applicants  with  higher  ability  in  the  Supervisor  category 

H4: 

Participants  will  bin  applicants  with  higher  benevolence  in  the  Co-Worker 
category 

H5: 

Participants  will  bin  applicants  with  higher  integrity  in  the  Others'  Supervisor 
category 

H6: 

The  above  posited  cultural  effects  will  be  greater  under  high  cognitive  load. 

H7: 

Interactive  behaviours,  such  as  speech  fluency  and  mouse  trajectories  are  likely  to 
change  during  the  high  cognitive  load  task  when  compared  to  the  low  load  task. 

Several  analyses  will  be  conducted.  First,  the  survey  data  will  be  aggregated  based  on  the 
pre-established  scales  used.  Reliability  analyses  will  be  conducted  to  ensure  that  these 
measures  are  reliable.  Various  analysis  techniques  (e.g.,  ANOVA,  regression)  will  be  used 
depending  on  the  hypothesis  to  be  tested.  Principle  component  analysis  will  be  needed  for 
the  survey  and  questionnaire  answers. 

For  Hypotheses  6  and  7,  a  number  of  features  of  interest  will  be  extracted  and  annotated.  The 
following  details  some  of  the  planned  feature  extraction  activities  on  each  of  the  behavioural 
measures  recorded. 

Speech  data 

1.  Data  cleaning  (e.g.  remove  cross-talk,  segmentation) 

The  speech  data  has  been  recorded  in  segments,  which  correspond  to  each  of  the  three 
subtasks.  Since  the  experiments  took  place  in  a  classroom  laboratory,  a  number  of 
participants  completed  the  sessions  at  the  same  time.  Although  directed  microphone  headsets 
were  use,  and  participants  were  seated  as  far  away  as  possible  from  one  another,  there  is  a 
chance  that  cross-talk  has  affected  the  speech  recordings.  It  will  be  necessary  to  clean  the  data 
by  extracting  any  noise  or  speech  from  other  participants  from  the  recording. 

2.  Build  CL  models,  test  data 

Some  of  the  data  will  be  used  to  create  a  low  load  and  a  high  load  models  of  speech  for  each 
of  the  three  tasks,  while  the  rest  of  the  data  will  be  used  to  test  the  models.  This  will  verify 
whether  cognitive  load  can  be  detected  from  the  acoustic  features  of  speech  in  this 
application. 

3.  Linguistic  analysis  of  think-aloud  speech 

Once  the  speech  data  is  pre-processed  and  cleaned,  mid-level  features  such  as  pause 
frequencies  and  lengths  can  also  be  annotated.  Linguistic  speech  features  can  also  be  collected 
from  the  transcripts  (which  themselves  can  be  generated  automatically).  Features  such  as 
frequency  and  type  of  pronoun  use,  sentence  complexity  (including  sentence  length  and 
average  word  length),  total  text  length,  use  of  affective  words,  use  of  cognitive  words,  among 
other  word  categories. 
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4.  Transcriptions  and  qualitative  analysis  of  speech  data 
Finally,  qualitative  analysis  can  be  useful  in  this  instance  to  further  understand  the  thought 
process  through  which  the  participant  arrives  at  their  response.  Similarities  in  thought 
processes  between  questions/sub-questions,  can  provide  more  information  about  how  trust 
judgments  are  made. 

Justification  Text 

The  justification  text  will  undergo  linguistic  analysis,  including:  frequency  and  type  of 
pronoun  use,  sentence  complexity  (including  sentence  length  and  average  word  length),  total 
text  length,  use  of  affective  words,  use  of  cognitive  words,  among  other  word  categories. 
These  features  will  be  used  for  comparison  purposes  between  the  low  and  high  load 
conditions. 

Mouse  trajectories 

Initially,  a  parsing  tool  will  be  built  that  can  display  each  trajectory  along  a  time  scale,  and 
allow  closer  inspection  of  movement.  This  will  allow  exploratory  analysis/  inspection  of 
mouse  behaviors  which  are  typical  of  this  application.  Some  basic  features  that  can  be 
automatically  extracted  from  this  dataset  include: 

•  Time  spent  moving  mouse 

•  Distance  traveled  per  task/  per  session 

•  Categorizing  time  spent  in  different  screen/window  areas  on  a  per-task  basis 

•  Which  areas  of  the  screen  were  most  frequented 

•  How  much  time  spent  on  specific  widgets,  e.g.  drop  down  boxes. 

•  Which  information  was  looked  at  when  answering  which  questions. 

•  Which  questions  were  hesitated  on/  Which  questions  they  were  much  more 
decisive  on 

•  Pauses  in  mouse  movement  indicate  thinking  -  this  will  help  to  identify  sections 
of  high  load. 

While  there  may  be  a  large  individual  differences,  the  trends  may  still  indicate  relative 
changes  at  different  points  in  time  during  the  task. 

Other  interactive  behaviors 

Application  level  behaviors  such  as  false  starts  in  answer  selections,  changes  in  selections,  etc 
can  also  give  an  indication  of  high  load  instances  within  the  session  or  task. 
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8.  Appendix 


Consent  Form 


UNSW 


£3 


NICTA 


UNSW  Approval  No  (08/2010/42) 
University  of  Sydney  HREC  Protocol  No  (14031) 


THE  UNIVERSITY  OF  NEW  SOUTH  WALES  (UNSW)  AND  NATIONAL  ICT  AUSTRALIA 

(NICTA) 


PARTICIPANT  INFORMATION  STATEMENT  AND  CONSENT 
FORM 


Applicant  Screening  Tool 


[Participant  selection  and  purpose  of  study] 

You  (i.e.  the  research  participant )  are  invited  to  participate  in  a  study  of  a  new  “Applicant  Screening 
Tool”.  We  (i.e.  the  investigators )  hope  to  learn  to  investigate  how  people  rate  applicants  using  a  newly 
developed  software  tool.  You  were  selected  as  a  possible  participant  in  this  study  because  you  are 
sufficiently  removed  from  this  area  of  expertise.  A  total  of  approximately  360  participants  will  be 
enrolled  over  several  phases  of  the  study. 

[Description  of  study  and  risks] 

If  you  decide  to  participate,  we  will  ask  you  to  interact  with  a  computer  simulation  and  answer 
questions  using  a  computer.  During  this  task,  you  will  be  asked  to  review  several  pieces  of  information 
for  numerous  applicants  and  rate  their  suitability  for  different  roles.  You  will  also  be  required  to 
complete  a  few  questionnaires  related  to  the  task.  We  will  also  be  collecting  speech  data,  mouse 
trajectories  and  keyboard  strokes,  but  this  should  not  interfere  with  your  ability  to  complete  the  task. 

We  will  also  train  you  such  that  you  are  confident  in  completing  the  tasks.  The  training  will  happen  at 
the  beginning  of  the  session  and  give  you  an  opportunity  to  familiarise  yourself  with  the  system’s 
functionality  and  the  tasks  to  be  completed. 

Your  participation  will  not  involve  risks  or  discomforts  greater  than  you  would  encounter  when  using  a 
computer. 

[Confidentiality  and  disclosure  of  information] 

Data  will  be  collected  from  participants  in  the  United  States  of  America,  Australia,  and  Malaysia.  Data 
will  be  shared  with  the  investigating  researchers  at  each  of  the  three  data  collection  sites.  Any 
information  that  is  obtained  in  connection  with  this  study  and  that  can  be  identified  with  you  will 
remain  confidential  and  will  be  disclosed  only  with  your  permission,  except  as  required  by  law.  If  you 
give  us  your  permission  by  signing  this  document,  we  plan  to  publish  the  results  in  academic  journals 
and  conference  proceedings,  as  well  as  to  build  models  on  features  derived  from  the  recorded 
behavioural  signals.  These  models  cannot  be  reversed  to  recover  any  of  the  original  signals,  and  the 
models  may  be  used  for  our  own  evaluations,  as  well  as  be  used  offsite  by  outside  third  parties  that  we 
would  grant  the  permission.  In  any  publication  of  results  and  distribution  of  models  or  related  software, 
information  will  be  suppressed  in  such  a  way  that  you  cannot  be  identified. 
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[Recompense  to  participants] 

You  will  be  given  course  credit  as  part  of  the  user  study  package  at  the  completion  of  the  study. 

Complaints  may  be  directed  to  the  Ethics  Secretariat,  The  University  of  New  South  Wales,  SYDNEY 
2052  AUSTRALIA  (phone  9385  4234,  fax  9385  6648,  email  ethics.sec@unsw.edu.au).  Any  complaint 
you  make  will  be  treated  in  confidence  and  investigated,  and  you  will  be  informed  of  the  outcome. 

[Your  consent] 

Your  decision  whether  or  not  to  participate  will  not  prejudice  your  future  relations  with  The  University 
of  Sydney  or  The  University  of  New  South  Wales  or  National  ICT  Australia.  If  you  decide  to 
participate,  you  are  free  to  withdraw  your  consent  and  to  discontinue  participation  at  any  time  without 
prejudice. 

If  you  have  any  questions,  please  feel  free  to  ask  us.  If  you  have  any  additional  questions  later,  Dr. 
Fang  Chen  on  (02)  9376  2101  will  be  happy  to  answer  them.  You  will  be  given  a  copy  of  this  form  to 
keep. 
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THE  UNIVERSITY  OF  NEW  SOUTH  WALES  AND  NATIONAL  ICT  AUSTRALIA 

PARTICIPANT  INFORMATION  STATEMENT  AND  CONSENT  FORM 

(continued) 

Applicant  Screening  Tool 


You  are  making  a  decision  whether  or  not  to  participate.  Your  signature  indicates  that,  having 
read  the  Participant  Information  Statement,  you  have  decided  to  take  part  in  the  study. 


Signature  of  Research  Participant 


Signature  of  Witness 


(Please  PRINT  name) 


(Please  PRINT  name) 


Date 


Nature  of  Witness 


Signature(s)  of  Investigator(s) 


Please  PRINT  Name 


REVOCATION  OF  CONSENT 

Applicant  Screening  Tool 

I  hereby  wish  to  WITHDRAW  my  consent  to  participate  in  the  research  proposal  described  above  and 
understand  that  such  withdrawal  WILL  NOT  jeopardise  any  treatment  or  my  relationship  with  The 
University  of  New  South  Wales  and  National  ICT  Australia  Ltd. 


Signature 


Date 


Please  PRINT  Name 


The  section  for  Revocation  of  Consent  should  be  forwarded  to  Dr.  Fang  Chen,  Locked  Bag  9013, 
Alexandria,  NSW  1435,  Australia. 
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Protocol  for  Training  Applicant  Screening  Tool 

Over  the  next  60  minutes  you  will  be  asked  to  complete  a  series  of  tasks. 

This  information  will  be  used  to  evaluate  a  new  applicant  screening  tool,  which  will  be  used 
by  human  resource  departments  and  potential  employers. 

The  design  of  the  tool  was  modelled  after  social  media  websites  to  standardize  the  hiring 
process  for  positions  with  large  number  of  applicants. 

The  tool  provides  employers  an  applicant's  virtual  resume  and  assist  employers  in  selecting 
the  best  fitting  applicants  for  various  positions. 

The  tool  includes  several  different  features  that  may  be  added  to  the  final  product. 

You  will  be  asked  to  complete  survey  questions  and  we  will  record  your  verbal  responses  to 
better  evaluate  the  tool  and  different  features  provided. 

Please  describe  the  steps  you  take  and  decisions  you  make  verbally.  Please  also  verbalize  any 
emotional  responses  you  have  to  the  applicants. 

A  demonstration  of  this  will  be  provided  in  a  training  video. 

Following  the  training  video,  you  will  be  presented  with  two  versions  of  the  tool  with 
varying  features.  The  training  exercise  will  begin  now,  which  will  outline  these  features. 

Script  Task  1:  Rating  Applicants 

This  screen  shot  shows  an  overview  of  each  applicant's  virtual  resume. 

Each  resume  provides  limited  information  of  the  applicant. 

At  the  top  of  the  screen,  moving  from  left  to  right,  you  will  see  a  tab  for  each  applicant.  There 
are  three  categories  of  information  for  each  applicant: 
education, 
experience,  and 

volunteer  experience  and  personal  interests. 

Within  each  of  these  categories  is  a  statement  about  the  applicant. 

As  part  of  the  audio  recording,  please  read  aloud  each  recommendation. 

For  example: 

After  reviewing  the  profile  and  reading  aloud  each  recommendation,  you  must  complete  the 
questions  listed  at  the  bottom  of  the  screen. 

You  will  complete  this  for  each  applicant.  Use  this  task  to  familiarize  you  with  the  applicants. 
After  completing  the  questions  for  each  of  the  four  applicants,  you  will  now  move  onto 
selecting  the  applicant  for  the  best  fitting  position. 

Task  2  :  Selecting  Applicants  into  Position 

This  screen  shot  shows  a  summary  of  the  information  presented  in  the  four  applicant 
resumes. 

Use  this  summary  page  to  select  the  applicant  who  is  most  appropriate  for  the  position  listed 
on  the  right  side  of  the  screen. 

This  includes: 

a  supervisor  within  your  department, 
a  co-worker  within  your  department, 
and  a  supervisor  for  a  different  department. 

You  will  not  be  able  to  select  the  same  applicant  for  more  than  one  position. 

To  place  an  applicant  in  a  position  drag  and  drop  the  resume  into  the  appropriate  slot. 

Please  note  that  you  can  change  your  response  by  dragging  a  different  applicant  in  a 
previously  filled  slot. 

Once  you  have  selected  an  applicant  for  a  position,  please  rate  how  confident  you  are  in  the 
decision  made,  which  ranges  from  very  high  confidence  to  very  low  confidence. 

Please  remember  to  verbalize  your  actions,  feelings,  and  thoughts  throughout  your  decision 
process. 
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For  example,  you  may  say:  <> 

When  you  click  on  the  continue  button,  a  pop-up  will  ask  you  to  justify  your  decisions. 

Type  in  your  justifications  and  click  save. 

Please  provide  a  specific  and  brief  explanation  for  your  decision. 

Task  3:  Ranking  Applicants  into  Position 

In  the  next  task,  you  will  need  to  rank  the  participants  from  best  to  worst  for  the  three 
positions. 

Do  this  by  dragging  the  applicants  to  each  ranking  option. 

Again,  you  can  change  the  order  of  the  ranked  applicants  at  any  point  before  clicking  the 
"continue"  button  by  dragging  a  different  applicant  into  the  ranking  slot. 

When  completing  the  ranking  be  sure  to  verbalize  your  actions,  feelings,  and  thoughts. 

For  example,  you  may  say:  <> 

Task  Summary  and  Scores 

Once  you  have  finished  these  three  tasks,  you  will  be  presented  a  review  of  your 
performance. 

This  review  covers  the  number  of  applicants  rated,  selected,  and  ranked. 

You  are  also  provided  feedback  on  your  accuracy  of  queuing  future  applicants. 

You  will  complete  two  full  sessions  of  these  tasks. 

You  will  use  the  full  version  of  the  tool  in  each  session,  but  only  one  session  will  include  the 
queuing  and  notification  feature  of  the  tool.  You  will  be  told  which  version  you  will  be 
completing  before  you  begin. 

Ok,  lets'  begin! 

The  Notification  Feature 

You  will  evaluate  two  versions  of  the  tool.  One  of  them  will  have  a  special  notification 
feature. 

You  will  notice  a  pop-up  in  the  bottom  right  of  the  screen  as  you  are  using  the  tool. 

This  feature  indicates  that  new  resumes  have  arrived  and  you  will  need  to  add  or  queue 
future  applicants. 

If  a  resume  is  listed  as  relevant  for  "Human  Resources,"  you  will  need  to  add  the  resume  to 
the  queue  by  clicking  on  the  box  and  then  clicking  "ADD." 

If  the  resume  is  listed  as  relevant  for  a  department  other  than  "Human  Resources,"  such  as 
"Finance,"  do  not  add  the  resume  to  the  queue. 

You  can  exit  the  queue  box  without  adding  an  applicant  by  clicking  the  "X"  at  the  top  of  the 
box. 

If  you  incorrectly  add  an  applicant  to  the  queue,  the  applicant  will  be  highlighted  red  to 
provide  you  immediate  feedback  of  your  error. 

You  will  only  have  4  seconds  to  decide  if  an  applicant  needs  to  be  added  to  the  queue  before 
this  information  disappears. 

You  will  be  scored  on  the  number  of  applicants  correctly  added  to  the  queue  so  be  sure  to 
carefully  attend  to  this  information  when  the  pop-up  appears. 


Pilot  Survey 

Thank  you  for  participating  in  our  survey.  The  survey  will  take  between  15  and  20  minutes  to 
complete.  The  survey  is  being  used  to  develop  and  validate  questions  for  a  study  on 
recommendations  for  job  applicants.  Participation  is  voluntary.  Data  will  only  be  used  for 
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study  development  purposes  and  no  personal  identification  information  will  be  recorded. 

At  the  end  of  the  survey,  you  will  be  asked  for  your  student  number.  This  is  so  we  can 
organise  to  get  your  Event  movie  voucher  to  you  in  appreciation  for  your  time,  and  your 
responses  will  remain  anonymous. 

If  you  have  any  questions  or  concerns,  please  contact  Dr.  Kevin  Eschleman 
(kevin.eschleman@wpafb.af.mil).  Thank  you  again  for  your  participation. 

1.  Please  choose  the  answer  that  best  describes  you. 

n 

Please  choose  the  answer  that  best  describes  you. 

Native  Australian:  You  were  born  in  Australia  and  have  lived  here  most  of  your  life  and 
identify  as  Australian. 

G 

Near-native  Australian:  You  were  NOT  born  in  Australia,  but  you  have  lived  here  for 
the  majority  of  your  life  and  identify  as  Australian. 

G 

^  Non-native:  You  were  NOT  bom  here  and  you  do  NOT  identify  yourself  as  Australian. 

2.  What  is  the  ethnicity  you  most  closely  identify  with? 

Rating  Applicant  Recommendations 

In  the  following  section,  you  are  given  a  list  of  recommendations  that  will  be  used  to  describe 
potential  employees.  The  recommendations  are  provided  by  coworkers,  supervisors, 
teachers,  and  friends.  Please  identify  the  personal  characteristic  you  think  the 
recommendation  describes.  Your  options  include: 

High  Ability:  The  recommendation  reflects  the  competencies  and  skills  relevant  to  being  a 
successful  employee. 

High  Benevolence:  The  recommendation  reflects  that  the  applicant  considers  other  people's 
interests  before  making  decisions  or  acting. 

High  Integrity:  The  recommendation  reflects  that  the  applicant  adheres  to  a  set  of  principles 
and  values  that  are  considered  positive. 

Neutral:  The  recommendation  does  not  strongly  reflect  any  specif c  personal  characteristic. 


Other:  Mark  "Other"  if  you  believe  the  recommendation  describes  a  personal  characteristic 
that  was  not  provided  as  an  option. 

3.  This  person  has  a  strong  moral  foundation. 

G  G 


Ability 

Benevolence 


Integrity 

Neutral 


Other  (please  specify)' 

4.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 

5.  This  person  is  inconsistent  in  performance  and  completion  of  tasks. 


5  -  Very  High 


Ability 

Benevolence 


G 

G 


Integrity 

Neutral 


Other  (please  specify)! 
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6.  Based  on  your  above  response... 


1  -  Very  Low  2  -  Low  3  -  Moderate 

7.  A  good  employee  whose  technical  know-how  is 

c  Ability  c 

F“°1  F^ 

^  Benevolence  “ 

4  -  High  5  -  Very  High 

widely  regarded  as  state-of-the-art. 

Integrity 

Neutral 

Other  (please  specify) ' 

8.  Based  on  your  above  response... 

1- Very  Low  2 -Low 

3  -  Moderate 

4  -  High 

5  -  Very  High 

9.  We  have  known  each  other  for  about  two  years. 
^  Ability  ^ 

Integrity 

n 

^  Benevolence 

□ 

Neutral 

Other  (please  specify^ 

10.  Based  on  your  above  response... 


1- Very  Low  2 -Low  3  -  Moderate 

11.  This  person  has  a  questionable  value  system. 

c  Ability  c 

F^ 

u  Benevolence  “ 

4  -  High 

Integrity 

Neutral 

5  -  Very  High 

Other  (please  specify) ' 

12.  Based  on  your  above  response... 

1  -  Very  Low 

2  -  Low 

3  -  Moderate 

4  -  High 

5  -  Very  High 

13.  We  have  similar  interests. 

^  Ability 

G 

Integrity 

□ 

Benevolence 

G 

Neutral 

Other  (please  specify)  T 

14.  Based  on  your  above  response... 

1  -  Very  Low 

2  -  Low 

3  -  Moderate 

4  -  High 

5  -  Very  High 

15.  This  person's  quality  of  work  is  not  adequate. 

^  Ability 

G 

Integrity 

Benevolence 

G 

Neutral 

Other  (please  specify) B 

16.  Based  on  your  above  response... 

1  -  Very  Low 

2  -  Low 

3  -  Moderate 

4  -  High 

5  -  Very  High 

17.  This  person  always  does  the  right  thing  even  in 

morally  difficult  situations. 

^  Ability 

G 

Integrity 

□ 

Benevolence 

G 

Neutral 

Other  (please  specify) B 

18.  Based  on  your  above  response... 

1  -  Very  Low 

2  -  Low 

3  -  Moderate 

4  -  High 

5  -  Very  High 

19.  This  person  needs  to  learn  to  consult  with  other  team  members  before  acting. 

^  Ability 

G 

Integrity 
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Benevolence 


Neutral 


Other  (please  specify)! 

20.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 

21. 1  am  surprised  we  have  not  interacted  more. 

□  Au;m..  G 


G 


Ability 

Benevolence 


Integrity 

Neutral 


Other  (please  specify)! 

22.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 

23.  This  person  is  not  honest  nor  upfront  about  intentions. 


Ability 

Benevolence 


Integrity 

Neutral 


Other  (please  specify)! 

24.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 

25.  This  person  is  currently  an  employee  at  our  organization. 

G  AU;1;i..  G 


Ability 

Benevolence 


Integrity 

Neutral 


Other  (please  specify)! 

26.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 

27.  Questionable  ability  to  perform  essential  job  tasks. 

G  AU;r;„,  G 


Ability 

Benevolence 


Integrity 

Neutral 


Other  (please  specify)! 

28.  Based  on  your  above  response... 
1- Very  Low  2 -Low 

29.  Please  mark  integrity. 

G 


3  -  Moderate  4  -  High 


G 


Ability 

Benevolence 


G 

G 


Integrity 

Neutral 


Other  (please  specify)! 

30.  This  person  enrolled  in  several  of  my  courses  while  in  school. 


Ability 

Benevolence 


Integrity 

Neutral 


Other  (please  specify)! 

31.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 

32.  This  person  does  not  meet  expectations  of  performance  in  all  areas. 

G  AU;1;i_.  G 


Ability 


Integrity 


5  -  Very  High 


5  -  Very  High 


5  -  Very  High 


5  -  Very  High 


-  Very  High 


5  -  Very  High 
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Benevolence 


Neutral 


Other  (please  specify)! 

33.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate 

34.  This  person  always  gets  things  done  well. 

□  G 


G 


Ability 

Benevolence 


4  -  High 


Integrity 

Neutral 


5  -  Very  High 


Other  (please  specify)! 

35.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate 

36.  This  person  has  a  tendency  to  ignore  others. 


Ability 

Benevolence 


G 

G 


4  -  High 

Integrity 

Neutral 


5  -  Very  High 


Other  (please  specify)! 

37.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 

38.  We  have  not  had  an  opportunity  to  interact  much. 

G  G 


5  -  Very  High 


Ability 

Benevolence 


Integrity 

Neutral 


Other  (please  specify)! 

39.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High  5  -  Very  High 

40.  The  work  devoted  to  planning  and  organizing  is  always  careful  and  complete. 


Ability 

Benevolence 


G 

G 


Integrity 

Neutral 


Other  (please  specify)! 

41.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 

42.  This  person  needs  to  improve  on  basic  work  skills. 

G  G 


5  -  Very  High 


G 


Ability 

Benevolence 


G 


Integrity 

Neutral 


Other  (please  specify)! 

43.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High  5 -Very  High 

44.  This  person  has  been  an  active  member  of  the  club  for  several  years. 


Ability 

Benevolence 


Other  (please  specify)! 

45.  Based  on  your  above  response... 

1- Very  Low  2 -Low 


G 

G 


Integrity 

Neutral 


3  -  Moderate  4  -  High 


5  -  Very  High 
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46.  This  person  is  one  of  my  most  fair  and  impartial  employees. 

I  li  i  f  in  n ..  mj-  m  \  .  immnt-  I"  m/4 


This  person  is  one  of  my  most  fair  and 
impartial  employees.  Ability 

G 

^  Benevolence 


Integrity 

Neutral 


Other  (please  specify)! 

47.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High  5  -  Very  High 

48.  This  person  does  not  demonstrate  a  commitment  to  helping  others. 

□  Au;m..  G 


Ability 

Benevolence 


Integrity 

Neutral 


Other  (please  specify)! 

49.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High  5 -Very  High 

50.  This  person  consistently  demonstrated  strong  skills  on  class  projects  and  understanding  of 
course  material. 


Ability 

Benevolence 


G 

G 


Integrity 

Neutral 


Other  (please  specify)! 

51.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High  5 -Very  High 

52.  The  interests  of  the  team  and  coworkers  were  always  placed  before  this  person's  self 
interests. 

G  AU;1;.„  G 


Ability 

Benevolence 


Integrity 

Neutral 


Other  (please  specify)! 

53.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate 

54.  This  person  is  not  always  truthful  and  honest. 


Ability 

Benevolence 


G 

G 


4  -  High 

Integrity 

Neutral 


5  -  Very  High 


Other  (please  specify)! 

55.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 

56.  This  person  is  proficient  and  compentent  while  completing  work. 

G  G 


5  -  Very  High 


G 


Ability 

Benevolence 


Integrity 

Neutral 


Other  (please  specify)! 

57.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 

58.  This  person  has  a  strong  character. 


5  -  Very  High 


Ability 


Integrity 
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Benevolence 


Neutral 


Other  (please  specify)! 

59.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate 

60.  This  person  is  always  excited  about  the  club. 

□  AUO...  G 


c 


Ability 

Benevolence 


4  -  High 


Integrity 

Neutral 


Other  (please  specify)! 

61.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 

62.  This  person  lacks  consistency  in  values  and  principles. 


Ability 

Benevolence 


G 

G 


Integrity 

Neutral 


Other  (please  specify)! 

63.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 

64.  This  person  does  not  put  in  an  extra  effort  to  help  others. 

G  G 


Ability 

Benevolence 


Integrity 

Neutral 


Other  (please  specify)! 

65.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate 

66.  This  person  performs  quality  work. 


Ability 

Benevolence 


G 

G 


4  -  High 

Integrity 

Neutral 


Other  (please  specify)! 

67.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 

68.  This  person  has  strong  values  that  are  well  respected. 

G  G 


G 


Ability 

Benevolence 


G 


Integrity 

Neutral 


Other  (please  specify)! 

69.  Based  on  your  above  response... 

1- Very  Low  2  -  Low  3  -  Moderate  4 -High 

70.  This  person  is  honest,  just,  and  impartial  without  fail. 


Ability 

Benevolence 


Other  (please  specify)! 

71.  Based  on  your  above  response... 

1- Very  Low  2 -Low 


G 

G 


Integrity 

Neutral 


3  -  Moderate  4  -  High 


5  -  Very  High 


5  -  Very  High 


5  -  Very  High 


5  -  Very  High 


5  -  Very  High 


5  -  Very  High 


5  -  Very  High 
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72.  This  person  consistently  completed  tasks  at  a  high  level. 

□  a  non..  G 


Ability 

Benevolence 


Integrity 

Neutral 


Other  (please  specify)! 

73.  Based  on  your  above  response... 

1- Very  Low  2 -Low 


3  -  Moderate  4  -  High 


5  -  Very  High 


74.  This  person  does  everything  possible  to  help  out  whenever  possible. 
^  Ability  n  ' 

n 

u  Benevolence 


n 

Integrity 
^  Neutral 


Other  (please  specify)! 

75.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate 

76.  Has  always  been  particularly  good  to  me. 


Ability 

Benevolence 


G 

G 


4  -  High 

Integrity 

Neutral 


5  -  Very  High 


Other  (please  specify)! 

77.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 

78.  This  person  takes  good  care  of  the  employees. 

G  AU;1;i..  G 


5  -  Very  High 


Ability 

Benevolence 


Integrity 

Neutral 


Other  (please  specify)! 

79.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High  5 -Very  High 

80. 1  was  always  confident  the  work  would  be  completed  efficiently  and  competently. 


Ability 

Benevolence 


G 

G 


Integrity 

Neutral 


Other  (please  specify)! 

81.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 

82.  This  person  is  kind  and  caring  toward  others. 

G  AU;1;i_.  G 


5  -  Very  High 


Ability 

Benevolence 


Integrity 

Neutral 


Other  (please  specify)! 

83.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High  5 -Very  High 

84.  This  person  goes  well  above  and  beyond  the  call  of  duty  to  look  out  for  other  students' 
best  interests. 


Ability 


Integrity 
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Benevolence 


Neutral 


Other  (please  specify)! 

85.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 

86.  This  person  puts  forth  an  extra  effort  to  support  others. 


5  -  Very  High 


c 


Ability 

Benevolence 


c 

□ 


Integrity 

Neutral 


Other  (please  specify)! 

87.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 

88.  This  person  has  been  part  of  our  team  for  several  years. 


5  -  Very  High 


Ability 

Benevolence 


G 

G 


Integrity 

Neutral 


Other  (please  specify)! 

89.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High  5  -  Very  High 

90.  This  person  is  a  valuable  employee  who  performed  at  a  high  level. 

G  G 


Ability 

Benevolence 


Integrity 

Neutral 


Other  (please  specify)! 

91.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 

92.  This  person  is  respectful  of  other  people  and  their  opinions. 


5  -  Very  High 


Ability 

Benevolence 


G 

G 


Integrity 

Neutral 


Other  (please  specify)! 

93.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 

94.  This  person  lacks  a  moral  center. 

G  G 


5  -  Very  High 


G 


Ability 

Benevolence 


G 


Integrity 

Neutral 


Other  (please  specify)! 

95.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate 


4  -  High 


5  -  Very  High 


96.  This  person  needs  to  become  more  compassionate  and  caring  toward  others. 

G  G 


Ability 

G 

Benevolence 
Other  (please  specify)! 


Integrity 

Neutral 
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97.  Based  on  your  above  response... 

1  -  Very  Low  2  -  Low  3  -  Moderate  4  -  High 

98.  This  person's  judgement  is  not  always  ethically  sound. 

^  Ability  ^  Integrity 

c  c 

Benevolence  Neutral 

Other  (please  specify) 

99.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 


^  Ability 

n 

Benevolence 


Other  (please  specify)! 

101.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 


102.  Please  mark  benevolence 
^  Ability 

n 

^  Benevolence 

Other  (please  specify)  T 

103.  This  person  acts  on  the  up  and  up  from  what  I've  seen  and  heard. 

^  Ability  ^  Integrity 

□  C 

Benevolence  Neutral 

Other  (please  specify)  T 

104.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 


n 

Integrity 

P 

^  Neutral 


105.  This  person  is  always  fair  and  honest 
^  Ability 

C 

u  Benevolence 

Other  (please  specify)  T 

106.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 


Integrity 

P 

^  Neutral 


107.  This  person  does  not  tolerate  or  listen  to  others. 


C 


Ability 


Integrity 


Benevolence 


Neutral 


Other  (please  specify)) 

108.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 


5  -  Very  High 


5  -  Very  High 


5  -  Very  High 


5  -  Very  High 


5  -  Very  High 


5  -  Very  High 
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109.  This  person  does  not  have  a  strong  s 


^  Ability 

n 

Benevolence 


Integrity 
Neutral 


Other  (please  specify)! 

110.  Based  on  your  above  response... 

1- Very  Low  2 -Low  3  -  Moderate  4 -High 


5  -  Very  High 
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Pre  -  Screening  Survey 

You  have  been  offered  the  opportunity  to  participate  in  a  cross-cultural  decision  making 
study  using  the  "Applicant  Screen  Tool."  The  purpose  of  this  research  is  to  investigate  how 
people  rate  applicants  using  a  newly  developed  software  tool.  Please  complete  the  following 
survey  in  preparation  for  the  experiment.  If  you  have  questions  or  concerns  regarding  the 
survey,  please  contact  Dr.  Natalie  Ruiz. 


Dr.  Natalie  Ruiz 
Natalie.Ruiz@nicta.com.au 
T  +61  2  9376  2160 

1.  Please  provide  your  student  ID  (unikey).  This  information  will  be  used  to  link  your  survey 
responses  to  your  participation  in  the  study. 

2.  The  following  statements  describe  beliefs  about  the  world.  Please  indicate  your  agreement 
with  each  statement. 


Strongly  disagree 

Disagree 

Neither  agree  or 

Agree 

Strongly  agree 

disagree 

An  individual  who  is  currently  honest  will  stay  honest  in  the  future. 

Any  phenomenon  has  numerous  numbers  of  causes,  although  some  of  the  causes  are  not  known. 
Everything  in  the  universe  is  somehow  related  to  each  other. 

We  should  consider  the  situation  a  person  is  faced  with,  as  well  as  his/her  personality,  in  order  to 
understand  one's  behavior. 

Nothing  is  unrelated. 

A  person  who  is  currently  living  a  successful  life  will  continue  to  stay  successful. 

Any  phenomenon  entails  a  numerous  number  of  consequences,  although  some  of  them  may  not  be 
known. 

Even  a  small  change  in  any  element  of  the  universe  can  lead  to  significant  alterations  in  other 
elements. 

The  whole  is  greater  than  the  sum  of  its  parts. 

Future  events  are  predictable  based  on  present  situations. 

When  disagreement  exists  among  people,  they  should  search  for  ways  to  compromise  and  embrace 
everyone's  opinions. 

Everything  in  the  world  is  intertwined  in  a  causal  relationship. 

It  is  more  desirable  to  take  the  middle  ground  than  go  to  extremes. 

Current  situations  can  change  at  any  time. 

The  whole,  rather  than  its  parts,  should  be  considered  in  order  to  understand  a  phenomenon. 

It  is  not  possible  to  understand  the  parts  without  considering  the  whole  picture. 

If  an  event  is  moving  toward  a  certain  direction,  it  will  continue  to  move  toward  that  direction. 

Every  phenomenon  in  the  world  moves  in  predictable  directions. 

It  is  more  important  to  pay  attention  to  the  whole  than  its  parts. 

It  is  more  important  to  pay  attention  to  the  whole  context  rather  than  the  details. 

It  is  desirable  to  be  in  harmony,  rather  than  in  discord,  with  others  of  different  opinions  than  one's 
own. 

Choosing  a  middle  ground  in  an  argument  should  be  avoided. 

It  is  important  to  find  a  point  of  compromise  than  to  debate  who  is  right/wrong,  when  one's  opinions 
conflict  with  other's  opinions. 
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We  should  avoid  going  to  extremes. 


The  following  statements  describe  individual  characteristics.  Please  indicate  your  agreement 
with  each  statement. 


Strongly  disagree 

Disagree 

Neither  agree  or 

Agree 

Strongly  agree 

disagree 

I  would  prefer  complex  to  simple  problems. 

1  like  to  have  the  responsibility  of  handling  a  situation  that  requires  a  lot  of  thinking. 

Thinking  is  not  my  idea  of  fun. 

I  would  rather  do  something  that  requires  little  thought  than  something  that  is  sure  to  challenge  my 
thinking  abilities. 

I  try  to  anticipate  and  avoid  situations  where  there  is  likely  a  chance  I  will  have  to  think  in  depth 
about  something. 

I  find  satisfaction  in  deliberating  hard  and  for  long  hours. 

I  only  think  as  hard  as  I  have  to. 

I  prefer  to  think  about  small,  daily  projects  to  long-term  ones. 

I  like  tasks  that  require  little  thought  once  I've  heard  them. 

The  idea  of  relying  on  thought  to  make  my  way  to  the  top  appeals  to  me. 

I  really  enjoy  a  task  that  involves  coming  up  with  new  solutions  to  problems. 

Learning  new  ways  to  think  doesn't  excite  me  very  much. 

I  prefer  my  life  to  be  filled  with  puzzles  that  I  must  solve. 

The  notion  of  thinking  abstractly  is  appealing  to  me. 

I  would  prefer  a  task  that  is  intellectual,  difficult,  and  important  to  one  that  is  somewhat  important 
but  does  not  require  much  thought. 

I  feel  relief  rather  than  satisfaction  after  completing  a  task  that  required  a  lot  of  mental  effort. 

It's  enough  for  me  that  something  gets  the  job  done;  I  don't  care  how  or  why  it  works. 

I  usually  end  up  deliberating  about  issues  even  when  they  do  not  affect  me  personally. 

Imagine  yourself  in  a  job  setting  and  rate  your  agreement  with  the  following  items. 


Strongly  disagree 

Disagree 

Neither  agree  or 

Agree 

Strongly  agree 

disagree 

It  is  important  to  have  a  good  working  relationship  with  your  direct  superior. 

It  is  important  to  be  consulted  by  your  direct  superior  in  his/her  decisions. 

A  subordinate  should  not  be  afraid  to  express  disagreement  with  his/her  superior. 

A  structure  with  a  subordinate  having  two  bosses  should  be  avoided. 

People  at  lower  levels  in  the  organization  should  carry  out  the  requests  of  people  at  higher  levels 
without  questions. 

People  at  higher  levels  in  organizations  have  a  responsibility  to  make  important  decision  for  people 
below  them. 

Once  a  manager  makes  a  decision,  people  working  for  the  company  should  not  question  it. 

In  work-related  matters,  managers  have  a  right  to  expect  obedience  from  their  subordinates. 

An  organization's  rules  should  not  be  broken,  not  even  when  the  employee  thinks  it  is  in  the 
company's  best  interest. 

Managers  should  make  most  decisions  without  consulting  subordinates. 

It  is  frequently  necessary  for  a  manager  to  use  authority  and  power  when  dealing  with  subordinates. 
Managers  should  seldom  ask  for  the  opinions  of  employees. 

Employees  should  not  disagree  with  management's  decisions. 
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Managers  should  not  delegate  important  tasks  to  employees. 

Most  organizations  would  be  better  off  if  conflict  could  be  eliminated. 

One  can  be  a  good  manager  without  having  precise  answers  to  most  of  the  questions  that 
subordinates  may  raise  about  their  work. 

In  order  to  have  efficient  work  relationships,  it  is  often  necessary  to  bypass  the  hierarchical  lines. 
I  am  uneasy  in  situations  in  which  there  are  no  clear  rules  or  guidelines. 

Conflicts  with  our  opponents  are  best  resolved  by  both  parties  compromising  a  bit. 


The  following  statements  describe  other  people.  Please  indicate  your  agreement  with  each 
statement. 


Strongly  disagree 

Disagree 

Neither  agree  or 

Agree 

Strongly  agree 

disagree 

Most  people  are  basically  honest. 

Most  people  are  trustworthy. 

Most  people  are  basically  good  and  kind. 

Please  identify  how  much  the  following  statements  describe  you  in  general. 


Strongly  disagree 

Disagree 

Neither  agree  or 

Agree 

Strongly  agree 

disagree 

I  am  the  life  of  the  party. 

I  sympathize  with  others'  feelings. 

I  get  chores  done  right  away. 

I  have  frequent  mood  swings. 

I  have  a  vivid  imagination. 

I  don't  talk  a  lot. 

I  am  not  interested  in  other  people's  problems. 

I  often  forget  to  put  things  back  in  their  proper  place. 
I  am  relaxed  most  of  the  time. 

I  am  not  interested  in  abstract  ideas. 

I  talk  to  a  lot  of  different  people  at  parties. 

1  feel  others'  emotions. 

1  like  order. 

I  get  upset  easily. 

I  have  difficulty  understanding  abstract  ideas. 

I  keep  in  the  background. 

I  am  not  really  interested  in  others. 

I  make  a  mess  of  things. 

I  seldom  feel  blue. 

I  do  not  have  a  good  imagination. 

7.  Please  identify  your  gender/sex. 

Please  identify  your  gender/sex.  Male 
Female 

8.  Please  identify  your  school/military  status. 


Please  identify  your  school/military  status.  College  Freshman 
College  Sophomore 
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College  Junior 
College  Senior 
Military 


Other  (please  specify)! 

9.  Please  select  the  response  that  best  describes  your  ethnicity. 


Please  select  the  response  that  best  describes  your  ethnicity. 

Near-native 

Native 

Self-identified  ethnicity:! 

10.  What  is  your  age  in  YEARS? 

11.  What  is  the  hand  you  primarily  use  (handedness)? 


c 

G 

G 

12. 


What  is  the  hand  you  primarily  use  (handedness)? 
Left 

Both  (ambidextrous) 

Do  you  wear  glasses  or  contact  lenses  to  read? 


Right 


Non-native 


G 

G 

13. 


Do  you  wear  glasses  or  contact  lenses  to  read?  Yes 
No 

Have  you  ever  been  involved  in  hiring  or  human  resources? 


G 

G 


Have  you  ever  been  involved  in  hiring  or  human  resources?  Yes 
No 
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Debriefing  and  Student  Exercise 

INF03315  -  Experiment  2  In-Class  Exercise 
HR  Applicant  Screening  Tool 

Debriefing 

Trust  is  a  critical  variable  in  military  operations,  be  it  trust  in  leadership  and  team 
members  or  between  opposing  sides  at  the  negotiation  table.  The  purpose  of  this 
study  is  to  examine  the  trust  process  across  cultures  and  establish  a  fundamental 
understanding  of  how  trust  operates.  The  three  data  collection  venues  include:  the 
U.S.  (Wright  State  University),  Malaysia  (Sunway  University),  and  Australia 
(NICTA) 

The  process  of  a  trust  judgement  has  both  affective  and  cognitive  components. 

The  HR  applicant  screening  tool  was  a  "cover  story"  in  order  to  get  samples  of 
people  making  trust  judgements  based  on  3  aspects:  Benevolence,  Integrity  and 
Ability.  The  applicant's  details  were  manipulated  along  these  3  variables.  In  the 
experiment  you  participated  in  (the  "Australian"  group),  the  cognitive  load  was  also 
manipulated  using  a  "dual  task"  design  (The  Notification  Feature).  This  was  an 
attempt  to  disrupt  the  trust  judgement  by  overloading  the  "cognitive"  processing 
and  forcing  subjects  to  use  more  "affective"  processes  to  make  the  judgements.  This 
would  either  change  the  applicant  you  trust  the  most,  or  not,  and  would  give  us  a 
better  idea  on  how  people  make  trust  judgements. 


Methodology 

1.  In  this  testing  experiment,  all  participants  completed  both  versions  of  the  software, 
with  and  without  the  notification  feature. 

a)  What  is  this  kind  of  design  called? 

b)  What  are  the  expected  benefits  of  this  design  type? 

c)  A  rank  order  effect  may  be  observed  in  this  experiment  design.  What  is  it  and  how 
can  it  be  counteracted? 

2.  As  a  tester,  you  were  asked  to  think-aloud  while  completing  this  experiment.  What 
kind  of  information  can  we  get  from  thinking-aloud  protocols? 

UI  Design  and  Implementation 

3.  The  tabbed  pane  widget  used  to  display  the  applicant  profiles  and  question  panels 
cannot  be  used  for  navigation  (to  switch  from  one  tab  to  another).  Instead,  the  user  is 
prompted  to  click  "Continue"  to  progress  within  the  task,  and  from  profile  to  profile. 
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j  4  Applicant  46-T 

1 - - - - - - - - - 

G 

Jj 

ar 


Applicant  46-T 

Project  Manager  at  Distal,  Xnc . 

Education 


Northern  University,  1998 

Bachelor  of  Arts:  Business 


.  "Judgment  is  not  always  ethically  sound."  -  Professor  of 
r  Business 


pjj  Experience 

Distal,  Inc.,  2003  -  Present 

Project  Manager 

•  Responsibilities:  Supervised  employees  on  product  development. 
Developed  a  model  for  employees  to  follow  to  increase  performance  and 
labor  costs. 

,  "Does  not  have  a  strong  set  of  principles.  Lacks  a  moral  center."  - 
Executive  Supervisor 

Arrow  Consulting,  1998-  2003 
Manager 

•  Responsibilities:  Led  consulting  projects  on  various  business  issues. 

Met  deadlines  and  provided  quality  products  to  customers. 


4? 


Volunteer  Experience  and  Personal  Interests 


The  Organization  for  Creative  Writers,  2003  -  Present 
Member 

•  Assisted  in  development  of  an  online  writing  portal  for  group  members  to  share  material  and  products 
j  "We  have  known  each  other  for  about  two  years."  -  Co-Club  Member 


The  applicant  has  specialized  capabilities  that  can 


Strongly 

Disagree 

—O 


Disagree 

—O 


Somewhat  Neither  Agree  Somewhat 
Disagree  or  Disagree  Agree 


Agree  Strongly  Agree 

_ 


The  applicant  is  well  qualified. 

■Hit;  jmaliir.anL  miultl  hfi  ymT  concerned  trtinur.  i 


_ 

k  1 

appli. 


1  desires  would  be  very  important  to  the 


a)  Cite  at  least  one  benefit  of  using  this  specific  design  combination. 

b)  What  is  its  major  drawback? 

4.  The  application  designers  wanted  to  highlight  some  specific  sections  of  the 
applicants'  profiles.  Cite  2  ways  in  which  they  tried  to  achieve  this. 

5.  What  other  interaction  patterns  could  have  been  used  instead  of  radio  buttons  to 
record  the  answers  to  each  review  question? 


Task  2:  Fill  positions  and  Task  3:  Rank  applicants 
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9g  Fill  Positions  1  Rank  Applicants  | 


Applicant  46-T  (Supervisor) 


Applicant  36-R  (Other's  Supervisor) 


^  Applicant  57-J 

'''"Acts  on  the  up  and  up  from  what  I've  seen  and  heard. 
Always  fair  and  honest  ".  Professor  of  Business  Finance 

'“'"Has  a  strong  moral  foundation.  Always  does  the  right 
thing  even  in  morally  difficult  situations.".  Executive 
Supervisor 

'''"Is  honest,  just,  and  impartial  without  fail.",  Co-Club 
Member 


Applicant  48-P  (Co-Worker) 


s|> 


& 


'^Please  justify  your  decisions: 


Drag  and  Drop  the  desired 
applicant  to  a  position 
listed  below.  Once  you 
have  selected  an 
applicant,  please  select 
the  level  of  confidence 
you  have  in  your  decision. 


Your  Supervisor: 


Applicant  46-T 


f Your  Co-worker : 


Other ' s  Supervisor : 

IE 


Applicant  36-R 


Cancel 


Save 


6.  The  layout  of  this  panel  changed  from  the  previous  task  panels.  Some  interface 
features  changed  and  some  others  stayed  the  same.  Why  do  you  think  that  was 
done? 

7.  The  drag  and  drop  mechanism  allowed  easy  placement  of  the  applicants  into  the 
ranking  slots.  Sketch  a  wireframe  of  an  alternative  mechanism  that  could  be  used  to 
achieve  the  same  goal  of  ranking  the  applicants. 
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8.  The  dialogue  box  to  "Justify  your  decisions"  popped  up  once  your  selections  were 
made,  which  was  not  a  good  way  to  solicit  feedback  about  the  rankings.  Give  2  ways 
in  which  it  could  have  been  done  better. 


Notification  Feature 


9.  The  notification  feature  allows  incoming  applicants  to  be  sorted  as  they  come  in,  at 
the  same  time  as  reviewing  applicants.  This  avoids  a  build  up  of  new  applicants  that 
need  to  be  sorted  through  later. 


,  Applicant  46-T  ^  Applicant  57 -J  Applicant  36-R  ^ 


j  Applicant  46-T 

Project  Manager  at  Distal,  Inc. 


(9 


Experience 


Education 


Northern  University,  1998 

Bachelor  of  Arts:  Business 


i  "Judgment  is  not  always  ethically  sound."  -  Professor  of 
r  Business 


Distal,  Inc.,  2003  -  Present 

Project  Manager 

•  Responsibilities:  Supervised  employees  on  product  development. 

Developed  a  model  for  employees  to  follow  to  increase  performance  and 
labor  costs. 


A 


"Does  not  have  a  strong  set  of  principles.  Lacks  a  moral  center." 
Executive  Supervisor 


Arrow  Consulting,  1998-  2003 
Manager 

•  Responsibilities:  Led  consulting  projects  on  various  business  issues. 
Met  deadlines  and  provided  quality  products  to  customers. 


Volunteer  Experience  and  Personal  Interests 


The  Organization  for  Creative  Writers,  2003  -  Present 
Member 

•  Assisted  in  development  of  an  online  writing  portal  for  group  members  to  share  material  and  products 
i  "We  have  known  each  other  for  about  two  years."  -  Co-Club  Member 


Task  List 

X 

Applicant  | 

Priority 

If  I  had  my  way,  I  wouldn't  let  this  person  have  any 
influence  over  issues  that  are  important  to  me. 

.ng  to  let 


control  over  i 


Somewhat  Neither  Agree 


.is  person 
■  future  in  this  compi 


I  would  wish  I  had  a  good  way  to  keep  an  eye  on  this 
person. 

Duld  be  comfortable  giving  this  person  a  task  or 

problem  which  was  critical  to  me,  even  if  I  could  not 
itor  their  actions. 


a)  What  interface  and  interaction  aspects  of  this  feature  worked  well?  (Name  one) 


b)  What  are  the  limitations  of  this  implementation  of  the  feature?  (Name  one) 


c)  How  can  these  be  overcome? 
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Training  Video  Storyboard 
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Applicant  Screening  Tool 

Task  1:  Rating  Applicants 

User  Testing 

Task  2:  Selecting  Applicants 
into  Position 


Task  3:  Ranking  Applicants 
into  Position 


2 


The  Notification  Feature 


Task  Summary  and  Scores 


Task  Review: 

Rating  Applicants 

Number  of  Applicants  that  were  rated:  4  of  4  (1 00%) 

Thank  you  for  ranking 
the  applicants. 

Please  click  continue 
when  you  are  ready  to 
take  some  surveys. 

Position  Selection 

Number  or  positions  mied:  3  or  3  (i  oo%) 

Number  of  Applicants  Ranked:  4  of  4  (100%) 

Cnfnra  Taelre  |M«t  1  leari  InThic  Coccinn) 

Manager:  Not  U**d  ln  Thl*  ®*sslon 

incorrectly  added  applicants:  Not  used  in  This  session 

Let’s  Begin! 


